{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering with RNNs\n",
    "\n",
    "There are many structured prediction tasks in machine learning, and many of them involve sequences - particularly sequences of text - in some form. Some examples include sentiment analysis (text to single class), image captioning (single image to text) and machine translation (text to text). Recurrent neural networks (RNNs) are a good fit for such problems, particularly when the sequences involved have an explicit or implicit ordering to the items; conversely, one might find other architectures more suitable if the input is a set. RNNs not only take in and output a single input and output at a time, but also have a hidden state vector which can be used to integrate information over time. They can have more complex units, such as long short-term memory (LSTM) units, that alleviate problems such as vanishing gradients, and be combined with other modules to enable bidirectional reading or attention or memory mechanisms.\n",
    "\n",
    "We'll focus on text-based question answering, using RNNs to read a story, a query, and predict the answer. This can, broadly speaking, encapsulate several natural language processing (NLP) tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "We'll use the first task from the 20 tasks of the bAbI dataset. This procedurally generated dataset was designed to test text understanding and reasoning, and includes tasks such as answering yes/no questions, counting items, performing coreference resolution, and even basic deduction. In each task there is a story to read, a question, and the answer, with 1000 examples for training and 1000 for testing per task. The first task is based on answering a question with a single supporting fact - we'll set up datasets to iterate over and show an example below. By the standards of the dataset, we'll be looking at a *weakly supervised* setting, as opposed to the *strongly supervised* setting where the indices of the supporting facts (out of all of the facts) are also provided during training.\n",
    "\n",
    "We'll use `torchtext`, which provides the dataset in a more convenient form. `torchtext` provides several NLP functions, such as tokenisation, as well as helpers for dealing with training on text data. There tends to be a lot of data processing for dealing with text, so it's worth consulting the documentation and other material for how to make use of the package. Here we'll get training and test datasets, along with metadata about the text (such as the vocabulary size)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import os\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import ticker\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.nn import functional as F\n",
    "from torchtext import datasets\n",
    "from IPython.display import clear_output, display\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Story:\n",
      "Mary moved to the bathroom\n",
      "John went to the hallway\n",
      "Daniel went back to the hallway\n",
      "Sandra moved to the garden\n",
      "John moved to the office\n",
      "Sandra journeyed to the bathroom\n",
      "-------------------------------------------------\n",
      "Query: Where is Daniel?\n",
      "Answer: hallway\n"
     ]
    }
   ],
   "source": [
    "def print_example(example):\n",
    "    story, query, answer = '\\n'.join(' '.join(s) for s in example.story), ' '.join(example.query), ' '.join(example.answer)\n",
    "    print('Story:\\n%s\\n-------------------------------------------------\\nQuery: %s?\\nAnswer: %s' % (story, query, answer))\n",
    "\n",
    "data_path = os.path.join(os.path.expanduser('~'), '.torch', 'datasets', 'babi')\n",
    "train_data, _, test_data = datasets.BABI20.iters(task=1, batch_size=32, root=data_path)\n",
    "STORY, QUERY, ANSWER = [train_data.dataset.fields[f] for f in ['story', 'query', 'answer']]\n",
    "print_example(train_data.dataset[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "We'll use a combination of models to deal with the different inputs and produce an output. As the words are symbols, these are first passed through an embedding layer to map each symbol into a (learnable) real-valued vector. For the story, we'll use a bidirectional LSTM, as it will be able to better preserve information across larger sequences (that are provided at once - only unidirectional RNNs can be used for online sequences). For the question, we'll use a unidirectional LSTM, and use the final output to \"attend\" to the sentence states of the story RNN (through a multiplication operation). Finally, this will be passed to a fully-connected network to predict the output (the answers are single words here, so there is no need for an RNN)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, vocab_size, hidden_size, zeros_idx, bidirectional=False):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, hidden_size, padding_idx=zeros_idx)\n",
    "        self.rnn = nn.LSTM(hidden_size, hidden_size, bidirectional=bidirectional)\n",
    "\n",
    "    def forward(self, x, h=None):\n",
    "        x = self.embedding(x)\n",
    "        if x.dim() == 4:  # Sum embeddings over a sentence in the story encoder\n",
    "            x = x.sum(2)\n",
    "        x, _ = self.rnn(x, h)\n",
    "        return x\n",
    "\n",
    "class QANetwork(nn.Module):\n",
    "    def __init__(self, hidden_size):\n",
    "        super().__init__()\n",
    "        self.s_encoder = Encoder(len(STORY.vocab), hidden_size // 2, zeros_idx=STORY.vocab.stoi['pad'],\n",
    "                                 bidirectional=True)\n",
    "        self.q_encoder = Encoder(len(QUERY.vocab), hidden_size, zeros_idx=STORY.vocab.stoi['pad'])\n",
    "        self.a_generator = nn.Sequential(nn.Linear(hidden_size, hidden_size),\n",
    "                                         nn.Dropout(0.8),\n",
    "                                         nn.ReLU(),\n",
    "                                         nn.Linear(hidden_size, len(ANSWER.vocab)))\n",
    "\n",
    "    def forward(self, x, h=None):\n",
    "        s = self.s_encoder(x.story)  # All hidden states\n",
    "        q = self.q_encoder(x.query)[:, -1]  # Final hidden state\n",
    "        attention = F.softmax(torch.einsum('bsh,bh->bs', [s, q]), dim=1).unsqueeze(2)  # Multiplicative attention mask\n",
    "        a = torch.sum(attention * s, 1)\n",
    "        a = self.a_generator(a)\n",
    "        return a, attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Testing\n",
    "\n",
    "We'll train the network for a few epochs and plot the training and test losses. We can also visualise the attention of the network over the story, showing which parts it thinks are relevant for answering the question at hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Final test accuracy: 85.60%'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0QAAAHjCAYAAAADn99RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzs3XecFdX5x/HvYVmaImVBQEBBjS0YQTc2NCZKbNgbGrGCilETMVFjjSViNBYULLHXnyUqEYnGiEajougCqyCIoiJVmlKl7LLn98fsMDP3zu137r27+3m/Xrxm5syZM88SDTyeOc8x1loBAAAAQFPUrNgBAAAAAECxkBABAAAAaLJIiAAAAAA0WSREAAAAAJosEiIAAAAATRYJEQAAAIAmi4QIAAAAQJNFQgQAAACgySIhAgAAANBkNS92AJnq1KmT7dWrV7HDAAAAAFCiJk2atNRa2zmdvg0uIerVq5eqqqqKHQYAAACAEmWM+TbdvnwyBwAAAKDJIiECAAAA0GSREAEAAABoshrcGiIAAAAA4WpqajRv3jytW7eu2KEURKtWrdSjRw+Vl5dnPQYJEQAAANBIzJs3T23btlWvXr1kjCl2OJGy1mrZsmWaN2+eevfunfU4fDIHAAAANBLr1q1TRUVFo0+GJMkYo4qKipxnw0iIAAAAgEakKSRDrnz8rCREAAAAAJosEiIAAAAAebFs2TL17dtXffv2VdeuXdW9e/dN1xs2bEhrjLPOOkszZ86MOFIPRRUAAAAA5EVFRYWqq6slSdddd50233xz/fGPfwz0sdbKWqtmzcLnZh599NHI4/QjIQIAAAAaoYsvlupzk7zp21caOTLz52bNmqVjjjlG++23nyZOnKhx48bp+uuv1+TJk7V27VoNGjRI1157rSRpv/320+jRo9WnTx916tRJw4YN02uvvaY2bdro5Zdf1pZbbpnXn4lP5gAAAABEbvr06RoyZIimTJmi7t27669//auqqqr0ySef6I033tD06dPjnlmxYoUOOOAAffLJJ9pnn330yCOP5D0uZogAAACARiibmZwobbfddvr5z3++6fqZZ57Rww8/rNraWi1YsEDTp0/XLrvsEnimdevWOuywwyRJe+yxh9599928x0VCBAAAACBym2222abzL7/8UnfddZc++ugjtW/fXoMHDw7dT6hFixabzsvKylRbW5v3uPhkDgAAAEBBrVy5Um3bttUWW2yhhQsX6vXXXy9aLMwQAQAAACio3XffXbvssov69OmjbbfdVv379y9aLMZaW7SXZ6OystJWVVUVOwxJUocO0tZbS598UuxIAAAAAGnGjBnaeeedix1GQYX9zMaYSdbaynSeZ4YoS+XlUm2ttHx5sSMBAAAAkC3WEGWppsY7f++94sUBAAAAIHuRJUTGmFbGmI+MMZ8YYz4zxlwf0qelMeY5Y8wsY8xEY0yvqOKJ0v77FzsCAAAAANmIcoZovaQDrbW7Seor6VBjzN4xfYZI+sFau72kOyXdEmE8eTd0aPB6u+0kY5xfAAAAAEpfZAmRdayuvyyv/xVbweFoSY/Xn78g6SBjGk468eCD3rkx0tdfB6/DtGzpJU0N5ycFAAAAGqdI1xAZY8qMMdWSFkt6w1o7MaZLd0lzJclaWytphaSKkHHONcZUGWOqlixZEmXIGWuW5HfQGOmZZ5zzv/zFud6wIb4PAAAAgOKINCGy1m601vaV1EPSnsaYPjFdwtKBuDrg1toHrLWV1trKzp07RxFq1jZuDF5bK91zj3f9m984n9Jdc02wX3NffT9/UjRnDrNHAAAAaJiWLVumvn37qm/fvuratau6d+++6XpD7MxAEo888oi+++67CCP1FKTKnLV2uaS3JR0ac2uepJ6SZIxpLqmdpO8LEVM+bbutc3S3dPrtb71zKfgpnbXOr5oaqdJXGb1lS+e4zTbRxgoAAABEpaKiQtXV1aqurtawYcM0fPjwTdctWrRIe5xCJkSR7UNkjOksqcZau9wY01rSAMUXTRgr6QxJH0g6QdJbtqHtFCvpq6/C260NzvTE/mQff+zd37BB+vWvg/dbt5bWrs1fnAAAAGhCLr5Yqq7O75h9+0ojR2b16OOPP6577rlHGzZs0L777qvRo0errq5OZ511lqqrq2Wt1bnnnqsuXbqourpagwYNUuvWrfXRRx9llExlKsoZom6S/muM+VTSx3LWEI0zxtxgjDmqvs/DkiqMMbMkXSLpTxHGUxTWSt9+G58M+e+7xo8P3lu3zjlefnmwEMOECdHECgAAAERh2rRpGjNmjCZMmKDq6mrV1tbq2Wef1aRJk7R06VJNnTpV06ZN0+mnn65Bgwapb9++eu655zKeWcpGZDNE1tpPJfULab/Wd75O0olRxVAqtt46+f2jjpLGjvWuv/02+OncrbcG+/fvL91/v3TeefmLEQAAAI1MljM5URg/frw+/vhjVdavGVm7dq169uypQw45RDNnztTvf/97HX744Tr44IMLHltB1hAhuZdf9s5btgwmUP5P7g45xDsfNiz6uAAAAIB8sNbq7LPP3rSeaObMmbrmmmtUUVGhTz/9VPvtt5/uvvtunVeE/+JPQlQi3GIL7mdyYf79b+nZZwsXEwAAAJAPAwYM0PPPP6+lS5dKcqrRzZkzR0uWLJG1VieeeKKuv/56TZ48WZLUtm1brVq1qiCxRfbJHHIT+9mcu9Zo0CDp5JOdc2Oc9m7dJH8RDmOkurrCxQoAAAAks+uuu+rPf/6zBgwYoLq6OpWXl+v+++9XWVmZhgwZImutjDG65RanBttZZ52loUOHFqSogmloRd0qKyttVVVVscMoiEQV6nbfXZoyJfmzl10m3RJb0w8AAACN2owZM7TzzjsXO4yCCvuZjTGTrLWVCR4J4JO5EuauKaupCbbXzyTGGTzYO48txAAAAAAgHglRCXv9dWdmqHnIh41lZd55v35OvyefDM4k+WeYAAAAAMQjIWqgamulESOcBCh2xujYY73zuXMLGxcAAACKq6EticlFPn5WEqIG7Iorwttfesk7T7UHEgAAABqPVq1aadmyZU0iKbLWatmyZWrVqlVO41BlrpHq2jVYeQ4AAACNX48ePTRv3jwtWbKk2KEURKtWrdSjR4+cxiAhaqQWLvTWED33nFOuGwAAAI1beXm5evfuXewwGhQ+mWsC3H2LAAAAAASREDVihxxS7AgAAACA0kZC1Ij9+9/e+bBhxYsDAAAAKFUkRE3E3/9e7AgAAACA0kNC1Mg9+2yxIwAAAABKFwlRI+evLteM/7UBAACAAP6K3AQ0ry+u7u7PZYz3CwAAAGjKSIiagJoa7zw2CSIpAgAAQFNGQoTAjNELLxQ7GgAAAKBwSIiaiPvvD15bK+21V3y/E08sTDwAAABAKSAhaiLOO885tmjhrSX68ENp40aprk7ad9/ixQYAAAAUCwlRE2KttH59sK1ZM+dTufff99rmzStsXAAAAECxkBAhTs+e2T334YesQwIAAEDDQkKETTp2zO35ffZxjqxDAgAAQENBQoRNli0rdgQAAABAYZEQIRT7EwEAAKApICFCQPPmxY4AAAAAKBwSIgTU1BQ7AgAAAKBwSIiQEOW3AQAA0NiRECGhbMtvAwAAAA0FCREAAACAJouECHE++KDYEQAAAACFQUKEOHvvXewIAAAAgMIgIUJSvXun12/gwGjjAAAAAKJAQoSkZs9Or98bb0QaBgAAABAJEiKEOvDAzPqzfxEAAAAaIhIihHrzzWJHAAAAAESPhAgpffhh5s+wqSsAAAAaAhIipNS/f+bPDBqU/zgAAACAfCMhQkp1dZk/M2VK/uMAAAAA8o2ECAl16pT9s2vX5i8OAAAAICokREhoyZJiRwAAAABEi4QIAAAAQJNFQoSMGOP8AgAAABoDEiKkpaIieP3QQ8WJAwAAAMgnEiKk5fvvpYMO8q7POcc79+851KpV4WICAAAAckVChKTKyrzzt94K73Pssd55nz7RxgMAAADkEwkRknrvvdR9Jk/2zkeNii4WAAAAIN9IiJDU3nun7uPfuDW2/447OkUYHn00v3EBAAAA+UBChKx9+GHqPl984RzPPjvaWAAAAIBskBAha/vsU+wIAAAAgNyQECEjc+cmvrfTToWLAwAAAMgHEiJkpEcP6Yorwu+98Ubm47HRKwAAAIqJhAgpPfhg8HrEiPB+PXpkNu6YMdnFAwAAAOQLCRFSGjpUstb5FauiIvtxhw3L/lkAAAAgH0iIkJPvv0987+ijkz+7eHF+YwEAAAAyRUKEvAibPRo7Nv3n58/PXywAAABAukiIkBW32lzr1uHJUJhkSU/fvrnHBAAAAGSKhAhZ6dHDSYR+/DH9Z444IvG9pUtzjwkAAADIFAkRCqa6utgRAAAAAEEkRAAAAACarMgSImNMT2PMf40xM4wxnxljfh/S55fGmBXGmOr6X9dGFQ8AAAAAxGoe4di1kv5grZ1sjGkraZIx5g1r7fSYfu9aa5OsLkFDt+WWlNgGAABAaYpshshau9BaO7n+fJWkGZK6R/U+lK5XXil2BAAAAEC4gqwhMsb0ktRP0sSQ2/sYYz4xxrxmjPlpgufPNcZUGWOqlixZEmGkyIdmMf9U7blnceIAAAAAUok8ITLGbC7pRUkXW2tXxtyeLGkba+1ukkZJ+mfYGNbaB6y1ldbays6dO0cbMHLWsWN2z40Zk984AAAAgFQiTYiMMeVykqGnrbUvxd631q601q6uP39VUrkxplOUMSF6J52Uus/tt8e3nXJK/mMBAAAAkomyypyR9LCkGdbaOxL06VrfT8aYPevjWRZVTIjY5MmSMbrn8H+l7HrjjfFt69dHEBMAAACQRJRV5vpLOk3SVGOMuyXnlZK2liRr7f2STpB0vjGmVtJaSSdba22EMSFKe+zhHI84QpLzP+NRRwW7fPSRs6ZoxYrChgYAAACEMQ0t/6isrLRVVVXFDgOJOBN+Gqb79HcNk/uPV32zOnWSlizxriXppz+VPvvMOW9g/zgCAACgBBljJllrK9PpW5Aqc2hC7r5bkjRKF+nXej3u9tKl8Y9MmxZ1UAAAAEA4EiLk10UXSTvvrCXqpBd1gtSiRdLurVoVKC4AAAAgBAkR8m/6dHUyi7S5Vks1NdLeeyfseuqpBYwLAAAAiEFChEi0qKvTpmVCEydqwNafhPZ76KGChQQAAADEISFCdHwVEl6f0zetR/zFFgAAAICokRAhWvVJUTNJF2pUwm4dOxYoHgAAAMCHhAjRe+opSdKdGq5f6U2NGRPfZZlvO15miQAAAFAoJESI3qmnSn36aLU21/MapPHH3RXajVkiAAAAFBoJEQpj6lT9R7+UkdW1GqEttDyuC7NEAAAAKDQSIhTMSfafOl2Pq0LL9JxOVgutjevDLBEAAAAKiYQIBfWqjtAw3a9D9bre0MFx9/2zRAAAAEDUmhc7ADQ9D2uodtF0XaI7nW/jfOW5AQAAgEJihggFdfXVzvFS/U0L1cW5SLBg6NFHCxQUAAAAmiwSIhTUjTc6xzqV6RT9n3ejWfw/imefXaCgAAAA0GSREKFo5m13oNS8/qtNa6Vddy1uQGkwhgp4AAAAjQkJEQpu3jypXTtp1ixJNTXejWnTpKef1v77Fy20tI0aVewIAAAAkA/GNrAF7ZWVlbaqqqrYYSDf/NMu1m66LLV/PGPCBAAAQAkyxkyy1lam05cZIpQGf3bBN2kAAAAoEBIilI5x4zadDtUDkqSPPgp2KeYanoULg9dduhQnDgAAAOQPCRFKx8CB0i9+IUkarYu0g2Zq33292/Pne+exiVIhHHRQ8Hrx4sLHAAAAgPwiIUJpeecdqbxcRlav6Ej13Thx060ePbxuxSi88Pnn8W2xs0YAAABoWEiIUHo2bNAM7aht9K2e1amhJd02bCh8WGFFFLbaqvBxAAAAIH9IiFCSdrNTdYFGa3t9Jf3ud0mnYsrKnF8AAABApkiIULIe1jm6V8MkSRu26pGwX12d86tQWrUq3LsAAAAQLRIilLQLNVrL1EHlqtMh+nfc/WHDvPORIwsT08knOxvLpsOtirf77tHGBAAAgOywMStKVkWF9P33Ulst1wztos21Rm/qlzpeL0ty1vTEluCO8h9n/2axgwZJzz+f+p1s5AoAAFB4bMyKRmHZMue4Su21jz5QrZrrOI3VtvpSknTKKcWLLZvZKPabBQAAKD0kRChp7qzKXG2j32q0JOl/OkDl2qBnn038nPupmvsr35OK3bpl99zRR+c3DgAAAOSGhAglr2dP5/i8nCmh7lqo0bpQUm1o/112iW/7+c8jCi5DY8c6CRr7FwEAAJQGEiKUvDlznJkiayVZq9VqrXP1oEbp93F9FyyQZszwrj/+2DsPS5QKJXb90FZbeZ8EAgAAoHhIiNDgbD7pPY3TQJ2v+3WYXg3c23pr7/zll6VK31I6f6KUqXzM6FgrXXaZd03lOQAAgOKjyhwapC3MCr2tX2l7zdIsbaM9NDWuj/8f7VyrvR18sPTGG8Hn/VXnEgnrQ+U5AACAaFFlDo3eKrXTURqr9Wqp3TVN48sHJO2/ww65ve/tt3N73q9Fi/yNBQAAgNyQEKHBmq8eGqKHJEkH1bypllq76d7OOwf7zpyZfKzy8viy2I8+Kl15pXNeU5NrtJ6JE/M3FgAAAHJDQoQG7RV5daz/qWMlOd+gTZ+e+JkFC+LbausL1g0c6LWdfbZ0882pP2t7+uk0g63Xt29m/QEAABAdEiI0SL/5jXPs0UObMpZD9br+oNtTPrvddonvvfpqfFvnzt558+bx96+6KuUrAQAAUKJIiNAgPf20kwfNnVvfMGmSlqqjbtVl+kB7Jn123brU4/s3XvWXx95jj/i+8+eHj3H55anfc+qpqfsAAAAgOiREaBx2312dThqgZrLaWx/HLwiS1LVr+KP9+8e3ffddeN8xY+LbasP3h03rU7r/+7/UfQAAABAdEiI0Hs89J222mXfdpUvgdqK9hCZMCF6HrTFy+WeOUlm8OP2+UfvHP5wcMSRPBAAAaNJIiNC4rF7tnS9enN53azG23TY/oSSrTPf73+fnHek666zCvg8AAKChICFC4+MvC3frraFTQ8lmgdavT+81YQUW0jVyZPbPZmPNmsK+DwAAoKEgIULj5E+Kttoq7nZYpbk//zm+LdG6I0nq3j2LuBLo0YPP2QAAAIqBhAiNl38aKCbbCKs0d911weuWLROvO5Kkm27KPjS/XXf1KtW1bZv586wNAgAAyB4JERqvbt2kM8/0ro3RllsGuyT7dC5Vee50S2a3aJH8/rRp3rl/CRQAAACiR0KExu3RR6V27TZdLqrpGLidbJPWfMn0HZnM9vjLg++4Y3rPfPppZvEAAAA0ZiREaPyWL/fOf/hBreVVGIidBWrVKv7x8vLcXn/VVeHtu+3mnS9cKO2wQ+Zjn322d/7FF+k9c8IJmb8HAACgsSIhQtPgK7Lwmg6TVKexY73bbiK0dq1zfPxx796GDdLddwfrNGQi0ad11dXO53SvveYUb5g507uX7izRm2+m7rNoUfD6q6/SGxsAAKApICFC01Gf0Rygd3WlbtbRR3u3/EmCtdLppwcfveiiaEJav1469FDv2r8/UTpJ0YYNwWv/J3Qu//iSVFeXfnwAAACNHQkRmpYFC2Ql3aSrdZxe2NQcUpm7KEaODCx5yrjqXFgp8Orq3GICAABozEiI0LR06yZzwQWqUZme1Onqp8nFjiiOf8lTplXnijH7Q9lvAADQkJEQoekZPVrlFe1Vpo0aqyO1Wx6SomT7FWUj0/VKffqk7tO5c+J73bpJzZtn9k4AAIDGgIQITdPSpWqpDdpKCzVZldJZZ+U03JVX5imuLL3xhnd+773hfaZOTfz8d99JGzdm/t4hQzJ/hhklAABQSkiI0HRZq2Zyfumxx3Ka5hkzJn9hxQorlBCra1fv3F+Ywa9Ll/B2/zqlQYPSj0uS/vGPzPoDAACUGhIiNG3+b9NyqKywYkUeYknAv9dQOmprM+vvX6f0/POZPbtqVWb9/W68MftnAQAA8oWECFiwwDsvwW+53nsvurFj9ygqpL/9rXjvBgAAcJEQAd26SZdd5l3nISmanMfidenOwrRsGd/25z+H93UTIf+ndrHctT4tWqT3/kzlMrsEAACQLyREgCTdcou05Zbe9eab5zTc8OE5xpOF44+Pb7v55vC+sWuF+vZNPG5NjZMY/f3vyd+/eHHy+wAAAKWIhAhw+b8fW7MmrQoDbdqEt0+ZkqeYkvj3v4PXTz8d36emJvzZ994LJjix8YZtCDtsWPJ4rrgi+X0pu6p0AAAAUSIhAvz8RRaefz7lt2/9+4e3Z7qhaphUX+6dd176Y5WXB683bkyc4Nx2WzD+009P7x1jx6bu88IL6Y0FAABQKCREQCx/UrTHHkm7Pv546iGy1b59fJt/D5+5c9MfK9nszZFHBq8vv9w7f/75xD9jrB9+SN1n5cr0xgIAACgUEiIgTJqV57p1iy6EsDVBfpkkXddf7xwrKuLvxc7s1NV55yeeGLzn3wA2VjYbuwIAABRbZAmRMaanMea/xpgZxpjPjDFx20Uax93GmFnGmE+NMbtHFQ+QkW7dghvlFKEcd7J9elIlS4ncd1/ie+l8Gnfwwdm9FwAAoFQZm49ve8IGNqabpG7W2snGmLaSJkk6xlo73dfncEkXSTpc0l6S7rLW7pVs3MrKSltVVRVJzECcHj2k+fOd81atpLVr47q4uZL/XyW3zZjgjEum/GNXV0v9+sX3adPGqQHh779woVNSO1lsklOqe9268Hv+51q08Ao0JBor9l6ynyed/l27SltsIX3xRfIxAQAAYhljJllrK9PpG9kMkbV2obV2cv35KkkzJHWP6Xa0pCes40NJ7esTKaA0zJvnna9bJx19dEaPh60DylaiWaFjjolvO/XU9Mb0J0OxDjjAOw9br/TWW+m9I5lEpbr3288p+vfll7m/AwAAIJmCrCEyxvSS1E/SxJhb3SX5/6o1T/FJk4wx5xpjqowxVUuWLIkqTCCcfxpj7NiMdl3df//8hTF7dnh7WLnt//1POuqo3N739tveeZcu8feHDs1tfEn6zW/C299/P/exAQAA0hF5QmSM2VzSi5IuttbG1pgKW5gR9xGNtfYBa22ltbayc+fOUYQJJJdG5bldd41vu//+/IWQyad3tbXSv/6Vv3f7VdZPPs+Zk/tY77wT31aE5VoAAKAJizQhMsaUy0mGnrbWvhTSZZ6knr7rHpIWhPQDii9F5blp0+IfibIKXSqpEqj//Ce7cSdNco75qCpXWxu8/t3vch8TAAAgE1FWmTOSHpY0w1p7R4JuYyWdXl9tbm9JK6y1C6OKCchJCVSeS6Vly/i2Fi2C1//5j1NR7te/ju/rToR98kn8vX32yT0+V6LfulGj4ttuvjl/7wUAAIgV5QxRf0mnSTrQGFNd/+twY8wwY8yw+j6vSvpa0ixJD0r6bYTxALm7+mppm22865YtQ/f2KZawwgt/+lPw+te/Tr7ZqrXSz34W3z5hQnj/Tp3Sj8+13XbJ75eVeed33ZX5+AAAAOmKrOx2VCi7jZLgm+JYttsv1OkTZzGM+69TWLnrXF5zzz3SBRc45y1aSBs2eH1i35FpKexs4vnuO6cstiTdeac0fHh673KfnzrVW3MVVsa7d2/pm2+8tlxKlwMAgKanJMpuA42a72/xFZ/8T81Um6Rz7m6/3TufGFursQi23947v/ji9J7xl+nu0yd539tu884b2H+zAQAADQwJEZAt39/UH9B5kb7KX9Gtb99IX5WW1aszf+bCC9Pve9xxwc/mAAAAokJCBOSivvLcED2ioXogstfEVmMrlmHDUvfxM8b7DC7TTVa7x+1IBgAAkH8kREAuunWT7r5bM/UT3aMLNd90KWqp7ajdd1/y+2GlxyWpR4/Mk7obbsisPwAAQDZIiIBcXXSRjtVLmq1e6qJl+vl3Ywr26lKbRfGvJ/InR/PnJ34mURJ1xhn5iUlyij4YI5WX529MAADQOJAQAXkwQ310pF6RldG9ukhttSKwZVG2YvcQclnr/Jo3L/d35NPHH3vnYaW7pfif6aijoovHNXKkcyyVTw8BAEDpICEC8qC8XPpCO+o6Xaeu+k7/0hG6ut+/ch53hx1yez5RQhWVVau880TV4fr1C17Pnh1ZOAAAACmREAF54JbFHqGrdJHu1v56TzriiJzH/dvfcnv+ootyDiFOspjSKZE9dmz4M1HEGuboowvzHgAA0DCwMSuQJ/4NUWdoJ+2kmc5Fjv+OZbrRqr9/VP96x74j9nrxYqlLl/Bn3ZhatvQ2mLVW6tBBWr482CffG9zGxgAAABonNmYFiuzBve7yLmL/Np6DZmn8G1voz+QkqXnz4HXPnqmfGT48eO0mQ2GWLMk8JgAAgHSQEAERuP3DQ6Q+fbyG2IwhS7vumrpPHr7US1uHDs6xU6dguzvzU1YmHXts+LN//Wv677n00sxjc02fnv2zmfDvuQQAABoOEiIgKlOnOhmBJG3cKO2+e85D/vvfqfu8+GLOr0lp2jSpTRvpu++c60QlshcskF56KbOxw2bBMh3Db//949sOPTT78VJZujS6sQEAQP6REAFR8td5njJFevrpnIbr2jX9vh075vSqpH76U2nNGu/zvESzPVtumfnYYRvb+qvXpeLuOeTO1nz/fXyf118Pf3aLLZznwpKodN16a3zbsmXOuHfemf24AAAgGiREQJ64f/mP23/Iv4J/8GDvb+s//3lksVjr/CW8GFq2zPyZxYu98+uvz+397p5Dsdq3l847L/mzbuL13nvZvz9shs79pPCSS7IfFwAARIOECMiTRYucROTqq0NuhpU1q6rykiNjQjKphsldP9S6dfrPDBzonQ8Z4p2Xl+cWy8sve+fvvy/df793PXhwbmMnMn9+NOMCAIBokBABhWKt8+uhh8LvX3ttMEGaNq2w8eXZjz965xdemLxvokr6O+2U2Ttvvjl4fcwx3vkuuwTvPf209PnnmY2fiH/dkJsQAgCAhoGECCi0IUO85Mha6aCDwvvtuqtkjO5RKUqHAAAgAElEQVTWhdpa3xY2xjwbNcr7cTN9LhNXXpm6jz+GnXfObPxETjwxfHxJ6t07P+8AAADRICECim38+GCCFFPD+iLdo2/VS4vVqSDrjwopVdGFAw7IbtwRI5Lf9+9rlM7eTql88EHie7Nn5z4+AACIDgkRUGqWLPGSo0WLdJoe1+90l5bJlyjFrj/605+KF2+IqVPT6/fmm/l7p3+90BVXJO/bqZNUUeGcuzM6uZTLXr8++2cBAEBxkRABpWzLLfWUTtco/U476/PE649uuaWk1h/596TNR79Urr46uF4oHbGV5H71q/zE4lesSn8AACB9JERAiXP3+jFG8euPDj88/KH69UcyxtlBtREzRrrpJu/aLa399tte209+Ev9cbMGGzz7Le2jq0iW+7ZRT8v8eAACQPRIioMStX+/kPnV1ITf/9a+k648kSWvXBmePfvazSOO9/PJIhw/Yb7/g9a67eqW1/euP3n8/9VixxRBy+YTOtXGjc9xqK69tzJjcxwUAAPlDQgQ0JjHrj1RWFt9n6tRI1h+5r/3rX/MyXBx/IQSXP9GxVvr00+D9bbd1fsTOnZOP/be/xbcde2zmMSbi35uoVNYbff+9tMcexY4CAIDiIyECGqstt5Rqa1Pvf1Ri648SGTQo8b3mzcPbv/oqwcxajKuuim+bODG9uNKVyUa1hVBRIU2eLHXtWuxIAAAoLhIioKloAOuPjEl8b8KExPfuvDO399bUpNeWi6OPzu94+bJoUbEjAACguEiIgKYqdv1R2KZAseuPdtkl72H4k6Bkn7Yl+9Tswguze/fBB8e3hX1l6GrWLPvfgmeeye45AAAQLRIiAI5Fi1KvP5oxI+/rj/yFEcL2DzrjjJxfkdDrr8e37bBDeN/LLnN+a2bMiC6efCkrk/7zn2JHAQBAw0BCBCBe7PqjF18M75eH9Uf/+593fvHF8fcfeyz8uenTM35VWl56KbzdX3jBLYUe2+6X62d82TLGWTd1yCHFeT8AAA0NCRGA1I47Lvh5XaISbBGvP/JXmttzz7wPLyl+f6Iw/vVFiRKiu+/OTzwAACBaJEQAMvfSS8EEKaxUWQTrj/zJypo1OQ+XE/fHSbRf0bx5iZ9dtiz1+O5vWyZ++cvM+gMAABIiAPmwcGHm648uumjTrWOPlQ46KPVrvv8+vm3rrXOIO4b/Uzi/LbbwzsvLnaO7lih2Q1dJGjfO+eIwkUQF/lynnJL8/qBB4WW833kn+XMAACAeCRGA/Ep3/dHo0ZuSo5fGGI2/8q2EQ553XuLXVVXlFq4/CTr33PA+q1Z5fTds8Nrvv987938heMst3nlYbjh5cvKYnn02+f3nn5fWrQsWTghLFgEAQGokRACile76o4MOchKkHXeMu+VPPGIlK9Wdjrvu8s5HjQrei/0cbv784PX553vn/h/rs8+88w4d4t+ZbPYoEyec4J136pSfMQEAaGpIiAAUVqr1R1984X1WF7LYxl9YIR+GDUt878QTg8mYm3Tcd19835EjvfOVK4NjuNq1yy7GRNyZK8n7dO+kk7y2WbMyG6+83Plt/8c/co+tUMrKpNmzix0FAKAhIyECUFz+9UetWgXvderk/A39kks2Ne2wg3TmmYUJbcKE4CyQKyyJ8s/QbNzond94o3f+5z+nfudVV6UfX5jnnvPOjz8+s2fdmauw8uelqF07p8R4797FjgQA0JCREAEoHWvXOolR7KKhO+/UDO2g7TRLy5dLTzyR39e6+Vgs/3qhTJMLV0WFdz58eOr+I0YEr7/4Inh93XXxzyRaP+T/dC8TDWU9kn+GDACAbJEQASg999/vZCi+RTw76kvN0k/0tn6h39gn1UwbMy5Lna5mIf/P+MILwet3343v07x5/mM566zgddi+R7vtFrx2f1/8M1XJxCZA69en91yxhSWxAABkioQIQOmqqNg0fWN69dIjOlM76gs9pdM1Vz21xraQ9tor76/t1St43a9ffJ/99otvy2SNUDp7EUnSJ58Er3/8Mb5P7J5H3bunH4cUv4ksiQYAoCkhIQLQMHzzjc62j6rrs05ZuK20UK1VI330kTMlEjatk6XHHw9epyqT7dp11/TfEVYV7qGH4ttSbUDr/4Suf3/n+PDDyZ/xl+uWpGeeSd4fAIDGLK2/QRhjtjPGtKw//6Ux5nfGmPbRhgYAIQYN8hb9+L9Rs9arThe7ECdD/tmfnXdO3M+teOd+ouYvoJDIwIGJ751zTurnY/l/1Pfec44HH5z8mdgqeXPnZv5eAAAai3T/k+qLkjYaY7aX9LCk3pL+L7KoACAdNTXhextddZWTpYRtApSh6dMT3+vUyXl9XZ1zHfYZXaxx47zzDz4I79O3r9S6dfJx3A1fa2pSvzNW7IzX2rXpFVJo1kw67bTM3wcAQClLNyGqs9bWSjpW0khr7XBJ3aILCwAy4O5tFLuT6vLl3qxRbLm2FBJVnsunfff1zv370U6Zkvrzu2yr3knxv02S9Kc/JX9mhx2c34+nnsr+vQAAlKJ0E6IaY8wpks6Q5P73zfJoQgKALPmKMGjLLYP3dtzRSYx+/evixOYTtreRm6+5M0P+jV7D+PcbylRYFblUm7F++WX27wMAoJSlmxCdJWkfSTdZa78xxvSWxH8nBFC6Fi1yEqM77gi2jx/vJEZR1MhO0733eucjRkjHHeddu1Xk9tkn/rnYYgjpCCvTHVaOe8WKzMcGAKAxMDbDb0KMMR0k9bTWfhpNSMlVVlbaqqqqYrwaQENXVuYt+PF78EFp6NC8vCJ2b6RE/xcbtoeSMcHw3D7uGF27Onme2+Yf46STgrNG7r327aUffkj97lixcfufKZWy3KUYEwCgNBhjJllrK9Ppm26VubeNMVsYYzpK+kTSo8aYO1I9BwAlZeNG52/Ov/pVsP2cc5y/XXfL79LIZJNQYX+BD8vV/NxkKEyiT+iWL0/8THmaHz5XpvXHSXG9/XaxIwAANFTpfjLXzlq7UtJxkh611u4haUB0YQFAhN56y8lIZs4Mtn/3nVeEId2dU5Po0SP5fXe50667Smeemf64qSrQSVLLlqn7tE9z84RJk9LrV0y33lrsCAAADVW6CVFzY0w3SSfJK6oAAA2bWzrNWqldu+C9Tp2cxOiUU7Ie/ne/S6/fp59Kjz6a/rgXXeQcW7VK3Oegg1KP06tX+u8sdelungsAQKx0E6IbJL0u6Str7cfGmG0lUXMIQOOxfLmTGF15ZbD92WedxCidKZcYw4fnKbYYt9ziHN0S2O6eRH5PPpl6nBNPzF9MxZbOPkoAAIRJKyGy1v7DWvsza+359ddfW2tz2AUDAErUTTeFb0K0YYP3Od24wk6UJ1ofdPzxTpi1tfH3OnZMPe6QIan7hFUp//rr1M8VWjYb1AIAIKVfVKGHMWaMMWaxMWaRMeZFY0yKr+MBoIFzE6O+fYPtRx7pJEbbbVeQMG64IZpx00maxo+Pb0v3U0AAABqCdD+Ze1TSWElbSeou6ZX6NgBo/KZMcRKjCROC7V9/ndciDInMmuVt3JqN00/P/JnYUt133umdf/hh9rHkyyOPFDsCAEBjkW5C1Nla+6i1trb+12OSOkcYFwCUnn328WaNYku91Rdh6K38f0+2YYO0227ZP5/OeqJY11wTvL74Yu+8FDZxzeZnAgAgTLoJ0VJjzGBjTFn9r8GSovvPoQBQ6n780UmMzj8/0DxTO+oODddempDgwfT5Nx5dt845JtvbKNbEidm/+623Et8LW7NUaLnMmAEA4JduQnS2nJLb30laKOkESWdFFRQANBj33uskRkuXSpKWqpMu0D36UP29z+k++CCrobfYIr5t7Nj0n99zT+989OjM3j1vXnxbixaZjRGl2E/6AADIVrpV5uZYa4+y1na21m5prT1GziatAABJqqiQrNV2rRdqa83RSzrGu7fvvk5i1K9fRkPut19822GHZReeu3dRun78Mb6tS5fs3h2F9euLHQEAoLFId4YozCV5iwIAGokrr5QWqauO1xjplVeCN6urvVmjNDz+eO7x3Hpr+n2b+f5E2Lgx/v5RR2UXQ6tWzo+8ww7ZPR+mrs45NsvlTzEAAJRbQpTen+gA0IRcfbVvG6MjjvAuYr83cxOjq65KOFZFRe7xXHqpd/7RR8n7dugQ3OR10qTg/RtvzC4Gdzbnywi28y6lz/gAAA1TLgmRTd0FACDJyQqslY49Ntg+YoSTGLVvn3KIXP/yv/fe8W3l5d75qadKbdp41xdcEOzboUNu749Cu3bFjgAA0NAlTYiMMauMMStDfq2SsycRACATL70UKMKwyYoV3qxRghJqb7+d3Ss328w52pD/jNWpk3d+111SD9+W21OnZve+QvrJT4odAQCgoUuaEFlr21prtwj51dZam0HxVwBAQH0RBlkrde0avLfjjk5i9MtfBpr32Se7V/krzD32WPDegAHB6wMP9M7DCitkaujQ3MdI5sgjox0fAND4RbYc1RjziDFmsTFmWoL7vzTGrDDGVNf/ujaqWACgpC1c6CRGDz4YbH/nHf1Lh2k3Vec0/JlneufPPBO8N3Jk8DpsnVAun+o9/HD2z6bj3HO98+rcfpsAAE1UlPV5HpN0aIo+71pr+9b/uiHCWACg9A0d6s0a1Vc3OEhvqVr99JV6Bxf8ZCn2M7iOHYPXYeuEttkm59duku/9g/xLr0aMyO/YAICmIbKEyFr7P0nfRzU+ADRqtbWStdpBM3WlbtLWmuO0GSM1z/6L5WwSkmvzOH9/XIQ72L3/fnRjAwAar2Lv4LCPMeYTY8xrxpifJupkjDnXGFNljKlasmRJIeMDgKJa2b6XbtaVWnHaeV7jxo1OYuSvkZ2mdevi2154QZo4MfEzgwfHt339dcavliS9+252z6Vj2bLoxgYANF7FTIgmS9rGWrubpFGS/pmoo7X2AWttpbW2snPnzgULEACK7YcfnC/oKp641zkZPty7WVeXdWLkd/zx0p57ZvbMFVek7nP11d65+7Vf2Iav+eLudwQAQCaKlhBZa1daa1fXn78qqdwY0ynFYwDQtN1xh5MY+TMSNzFqlvj/0k3MVtpJuqYlnRLgN9/sncduv5RPbM4KAMhF0RIiY0xXY5w/oo0xe9bHwgcPAJCOESOcxOj66702a729jGL06xe8zjWJSGctUl2dc+zeXXruudT9mzVzQv/XvzKLpaIis/4AAPhFWXb7GUkfSNrRGDPPGDPEGDPMGDOsvssJkqYZYz6RdLekk60N2zYQAJDQtdc6idAttwTb3cRo+XJJ0htvBG+HVZPLRE1N+OvCpLvBq/snQKZ7C+29d2b9AQDwMw0tB6msrLRVVVXFDgMAStM990gXXhjf/sMPMh28GtUHHJD4s7fYxMb/x0SLFl4y5La//LJ0zDHxfd1x3LbY62TvTfVHk3+s6mpvBizVc82aOfvgLliQvB8AoGEzxkyy1lam07fYVeYAAPl0wQVOVjB6dLC9QwedqiclORmDf7PWTIR9nuYmQ5LUv79zTFaF7ptvsnt3In37ptevQwfnt2bhwvy+HwDQsJEQAUBj5CZGDz+8qekpna4PtI8O1zideczyrIYdMCD5/QkTnOM++yTuM3Bg6vfEbhjrl05Bh1grVmz6ejBOss/9AACNHwkRADRmZ5/tJEbPPKMHNURdtEj/0pHOdIlvjVG67r47eO2W0461eLFzDEs0Pv889XuSFW24997Uz8dq3z68/YEHMh8LANC4kBABQFNw8sk6Vw9pB32h38r3OV2KxCh2i6PYYgy1tc4x0azP4Yd7523aOMdcl65muoz0xRcT33v88dxiAQA0fCREANBEtGkj1apc9+kCady44E03MVoW3P0g2V7YW2/tnccOF9Z+5ZWJx5o9O77ttNOC127OtnRp4nHCnHBC4nvpzFYBABo3EiIAaCKuu853MXCgM1Xz3/8GO3XqpCc0WEbOJkJDhiQeb+5c55gsafK76irvPKYquC65xLvnbhr71FNeW/PmXs7244+J33HxxYnvhU2CrVqVOm4AQONGQgQATcSll4Y0/vKXcYnRaXpa1eqrk/Ss/jJ0dspx3fVCLn/VuVR693aO/kIJb77pnRvjrP/ZuNFrc8/D1ifFri/yrx1q1847X7HCOcbupwQAaHpIiAAAXmI0ZYreVX+Vq0bP6RQnYzHG2ewnxFdfxbe9/HLi11gbXEPkztq4CYobSqVv5wj/Pb/mzb3z7t2dY2yC4z77i18E2++7L3GMAICmhYQIAJqQ2IQkTt++OqX7e+qjaTpHf/fa+/WLS4zat5e23VaBa79kpaxj+9Y5X+htSnI+/lg68UTvflg1u802887dct9+r7/unb/zTvDes8/G9//rXxPHCwBovEiIAAABfftKdSrTQzo3fhfVfv20QF3Vu9XcuNLY48cHrw86KPE7zj8/vN2/Hun556UpU6Rjj5U2bJAGDw729Rd18J+7Dj008fu//Ta+zbdlEwCgCSEhAgAEDB/uu+jVy5lS8iVG3bRIX6/b2pkC8i3+2WOP4DhvvJH4HSNGhLfHrj/q21d66SXn/Mkng/eSJVx+774b3xZWTGHevPTGc+2/v/NbsMMOmT0HACgtxua6IUSBVVZW2qpMN6EAAGTE/dwt7o+I2bO9Sgh+48ZJAwcGPpNL9ceL/x3u+Q8/JN5E1f+M5ORovXrF3zvwQGem6bnn4uNo3twryuB/b7oxJ4qlgf1RCgCNnjFmkrW2MnVPqXnqLgCApqaiQjr55JAb7ozR8uXBXVqPOEKS1EqrtU6bhTyYnmTJkOSsG1qzxgslzH//6yUoLVoE722+eeIiDQCApolP5gAAcZYulUaPTtKhfXsn64hZSLREXTRUD2pzRZN1JPusbbfdnKN/tia2JHjPnvmPCQDQsJEQAQCyF5MYlWuDHtS5+l6dnG/KHnkk5RD+PYjSeV0isZXkpODeQ5J0/PGp37FihRN6uhvOAgAaNhIiAEDu6hOjlj84UzLlqnXahwxJmRidc05mr0pUOjw2+Yn9XE6KKRiRwM9+5hyXLk0/ptgNYQEADQcJEQAgfxJ8SrcpMbrnnrhHYit750vs53JSfNIUZu7czN91+eWZPwMAKA0kRACA/EuUGF14oZMY3XqrWrZ0mtyqb/mWTvITJpuKcatXZ/cuAEDxkRABAKKTKDG6/HJ9s76rWuvHvL5uiy2c4+abp+4bu68RAKBpIiECAETPTYx8m/900yJ9rW11iy5TG63Jy2tWrHBeEbbxaqyHHsrLKwEADRwJEQCgsOrqNiVGX2gHXaa/6Rv11jvaX7rkkoKF8dlnqfsk2rNo5cr8xgIAKB4SIgBAcdTV6QD9T/vqfU1RP/1C70l33unMIP32t5G/PiypGTkyeH3BBeHP3nZbfNubb+YeEwCg8IzNZvVoEVVWVtqqqqpihwEAyIP6r+ckSa/qYB2mN4Idzj5bevjhyN4pSWVlXmGHjh2l77/37rVvH7/8SZK231766qtgW+fO4ZXtAACFZ4yZZK2tTKcvM0QAgJIw9Zb/OJ/SlZV5jY884mQwp50W2Xtbt/bO/cmQlPiTuQULvPNWrZzjkiX5jasQxo93fnvdYhQA0BSREAEASsJll9Wf1NY6iVHz5t7Np55y/uZ+0kl5f2/PnlKzBH8aJvqIYt0659i8uXTppXkPqWCuuso5plOEAgAaKxIiAEBpqqlxMpLycq/tH/9wEqPjjsvba446SmrbNtjWpUvyZ9xEqWNH6YYb8hZKwS1aVOwIAKD4SIgAAEWTaGYmYMMGJwNxd3KVpDFjnMRo4MCcY7jiCumnPw22zZyZ3rN77pnz64sq0SeBANCUkBABAIqma9cMOq9b5yRG7qIdSXr1VScxOuOMrGNo10669tr4tnTcckvWry0Ja9cWOwIAKD4SIgBA0dx+exYPrV3rJEb+aghPPJF2ue4WLeLbDjkkizgk7bJL8HrevOzGKZbaWu/8u++KFwcAFBMJEQCgaE4+OYeHf/zRSYw6dvTa7rvPSYyGD0/4WJQV1c45J7qxo1BX553n9L8FADRgJEQAgKKyNnE1t7QsW+YM0L691zZypJMYXX55XPfYWZ18eu+96MaOgv/3/aOPihcHABQTCREAoHH44Qfnb/ibb+613Xqrkxj5FgkNHZrZsA88kLqPWwhv9erMxi4lrCcC0FSREAEAGpdVq5zEaLPNvLYbb3QSo9tuS7nHa+wndTfdlPqV22+feZiSs+bImPiy3wCAwiEhAgA0TqtXxxdfuPRSyRi1VuKpnNjlR/Pnp37VqFHZhbjtts6xIc8sAUBDR0IEAGjc3OILvn2MvlVvXaw71V7LNrW1aeMcr7vOObrVvTduDA53xx3xrzjooOxCq6nJ7jkAQP6QEAEAmgZ3H6MWLfSVttOdukRztY3zzdrTT2vNmmCRgZ/9LHyYRx/NTzgrV+ZnHABAbkiIAABNy/r12kcf6hd6Rz+ovjLd4MFOYvTii5u6PfRQ+OPffJOfMLp0yc84+cSnewCaIhIiAECT9K5+oa01Tyor8xpPOMFJjF57Tbvu6jXffLPTfPjhzhd4UvCxbKxbl9vzUTjzzGJHAACFR0IEAGiyysok1dY636/5M5zDD5eMUQs5WcuVVzrNr73mfVbXrl34mGPHpn5vlJ/LnXGGdO650tNPp/+MMc7xjTfC719ySVz1cgBoNEiIAABN1qYCdG3beolRM++PxtnqrbP1sJprQ9yzu+8ePuaf/uQcjQn+6tzZ69O9e/xz6SRSySxY4LzniSekBx/0vgJMR0WFc1y1Kvz+nXc6x3RKkANAQ0NCBABosnr2jGlo29YpK7dypWSMflRrPayh+kEd9Jl2Upm8snA33hg+5qxZ0rPPxrcvXepUrisv99bqHHGEd/8vf8n+53jxxfAkK13u3kz+ohJh6uqyfwcAlCoSIgBAk3XFFQlutG0r1dVp1DmfaIL21Ob6Ubtopuarh07TEypTjfbeO/4RySmlPXiw126tl3itX+9MREnO7M0rr3j9ZszI/uc44YTg+/yJzcyZqZ8PKyUOAE0FCREAoMlxkwZ3ZiSRkQ+01b524qYZo3Zarid0hmZqJyej8WUbBxzgPefuXXT66c5xzhxpv/28+9dc4822NG/uHNesyfGHUvgMT//+uY8LAI0ZCREAAKnUzxi1+m6O/qxr1bK+2IJ28hKjsCIGjz/unb/7rpeI3XCD196xo3NM9blaImecEd6+1VbOcdmy8PvpohQ3gMaOhAgAgHR16aLr7fXqMf/jYPtOO2mLdkZGGzMe8tBDcwvJ3ToptoDCxx/H903Hd98Fr4cOzW4cAGgoSIgAAMjUVls5Uzrz5weav9COOkHPyyj96gOjRuUWivupnb+KnRtiNg47LHgdW/0ubMaoVSupT5/s3gcAxUZCBABAttzE6PPPJUkttU7/0CBNVj/NUffEdax9ttgiP6EkK4zw2mvxbePHB6/dEuSffhpsX7s2eB02Y7R+vfTZZ6ljBIBSREIEAECudtxRsla99bUG60l10HL11AIn22nWLK3EKFennpr43oknxre9+mrw2i0wkai0tvtJ3uuvB9urq73zRJvVAkApIyECACBPNqqFntZg/US+WtfWFjQx8tt5Z+cYVsFu8uTg9d//nnwsdyZrxYpg+x//6J2vXJlZfABQCkiIAADIs1rTykmE3n3Xa3QTo7KygiVGEycmvjd3bmZjuWXDY6vhTZoUvN5zz8zGBYBiIyECACBPbr/dOU6bVt+w335OBvH2216nurq8JUbDhye/724WG8ad6YmtTid5hRNmzfLaHnoofBx3VqhZ/d8osq1uBwDFQkIEAECeXHKJk//sskvMjQMOcG74F+64iVHz5lJ9ue5775WOOcZJUtwNW5NJlKSEOeWU4PWPPzrHsrL4vkcc4RyPOcZr69o1fFx3zVFlpdf27LPpxwUAxUZCBABAoRx2mJMYvfCC17Zxo+aol36hdzR6tPTyy5uaU3Jncjp0SNynosI5xiYpNTXOsbzca3PP33/fOdYXz9tUgS6Zm27yZpuuvjp1fwAoFSREAAAU2vHHO4nRU09Jkjppqd7RL/XejA46RU9lPNyVVya+t3Spd+7fdNVNuPzJzuGHO8fa2mAftz2ZAQOkzTZzzufMSd0fAEoFCREAAMVy6qmStfqZPtXvNVJttFb/p9P0hgboCL2c9jD+Sm/J9OzpnbvFEfz7ID2VIBd77LHgtT+x8vv5z52jO/sEAA0BCREAAEVW2+snulu/19aarT/oNu2qqXpF9YuJWrXKeXz3Mzx35sevRw/vfPPNvfPbbgtvl6Rhw8Lfc/fd2cUHAMVEQgQAQJGddJJzXKKuukN/0Haapat0o9O4fr2TGMVmJRk46ijvfMCA4L24AhD1Lr3UOfo/qXPXCPmL5vn16ZNVeABQVCREAAAU2VVXBa/XqK1G6GpNOO0OX+MaJyOpr6V9002ZvWOnnZzjm28G2wcOTP6cW41Oktq1c45uqW23qAMANGQkRAAAFJl/HY/fuZOHO4t9rrnGa1y9WjJGra/+XUbvmDEjvP3AA4PX/nLfzz8fvHfIIc7RXX+U6NM5AGhISIgAACgx7ianX35Z33DDDU4Wctllm/pcolF6UcfpLmWWGMWK/RLviiucY8eO0oknBu/F7ns0fnzicb/5JqewAKBgjHX/M08DUVlZaauqqoodBgAAeeWuz5GcvYOWLXPOQ/+YvuAC3X1vM52lx9RWq72H/DW2k7zD2uB5NnFa6+xbVFsrtWwprVsXvD9woDRuXGZjA0C+GGMmWWsrU/eMcIbIGPOIMWaxMWZagvvGGHO3MWaWMeZTY8zuUcUCAEBDcfTR0nHHJe+zz+R79HuNUh996jUuW+ZkI126RBugj1u1bpttvDb3k7t33y1YGACQkyg/mXtM0qFJ7h8m6Sf1v86VdF+EsUaW4i8AACAASURBVAAA0CD885/S7bcn7/Phh86xrkdvZ6rmN7/xbi5e7CRG3bsnfD4fxRAGD/bOL7zQO+/a1TmuWpX7OwCgECJLiKy1/5P0fZIuR0t6wjo+lNTeGNMtqngAAChl1nqfr9UXkttk9Ggnx/FvrCpJc+fWnzz9tPPw8cd7Nxcs8KrS+UvFSfrLX7KP0y3D/cwzXttFF3nnQ4Y4xwb2RT6AJqyYRRW6S5rru55X3xbHGHOuMabKGFO1ZMmSggQHAECpcBOOefOCa43ivPCCk4kccYTXtnq1tNlmkjHaRl9Lkp57LvtYHn/cOdbVhd+/7rrsxwaAYihmQhT2f+mh/z3JWvuAtbbSWlvZuXPniMMCAKD07bZbkpuvvOIkRi+9FMigZmkHjdKFOnL2nVm/N7byHAA0dMVMiOZJ8k/+95C0oEixAABQkhb4/mT86U+98+rqNB4+9lhnKmfNGqlzZy1VR52jB3W3LtF72lcXaJT0+ed5jxkAGpJiJkRjJZ1eX21ub0krrLULixgPAAAlZ6+9vPNp05z85YUXMhykTRtp8WKNuGixemientfx2kZzNFq/k3be2ZlFOu20tIdrlsHfHs4801t3BAClKMqy289I+kDSjsaYecaYIcaYYcYYd1/rVyV9LWmWpAcl/TaqWAAAaKjmzXOObhKy447B2gmZGDFCWqrOGqQXtI1m6/e6w7v51FNOYtShQ1wRhlinnJLe+x54wFlztG6ddMwx2cUMAFFjY1YAAEpQ27bB8th/+Yt01VW5jxtblMFaSf36hX+D99hj0hlnpBwn9q8SiQo/uP3atZNWrqQSHYDolMTGrAAAIHv9+wev85EMJTRlipOdPPZYsP3MM53spl+/hI+6G7H6tWmT+FWLFzvJkOQsbQKAYiMhAgCgBD3ySBFeesYZTmK0Zo3Uvr3XXl3tJEYtWkhz5gQe2XHH+GH69vXO/euNDjwwuF/seeflKW4AyAEJEQAAJWirrYr48jZtpB9+cJKjwYO99poaaZttJGO06uzz1aWLU+gh1h2+pUkrV0q77OKc//e/Um2td+/VV6MJHwAyQUIEAECJGziwiC9/8kknMZo0KfB93OaP3K/vFhlpyy3jijDstZczM7Tvvs6esJ99Fj708uVRBg4A6SEhAgCgxI0bF824iYofhNp9d2eGyFppp5289iVLnKynWTNpzJhNzRs3Su+/Hz7UIYc4R4oqACgFJEQAAJSoF15wqsvlk39NT0YJkd+MGU42M3q012atdNxxzqCxFSEU3E/p3//O8r0AEAHKbgMA0IRssYW0apVz3qKFtH59Hgb98UepS5dgnXBJatnS2UipUydJUrdu0qOPSoce6iVjDeyvIXnRpo0z2VZTU+xIgMaLstsAACDUzjt7561b52nQNm2cLMta6cgjvfb166XOnZ3s509/0sKFTjLUWO29d3gZ8lhr1waLSwAoLhIiAACaEP9+RltsEcELxo51EqP33w9+n3fLLU5iFFM+b/HiCGIokokTnbVT115b7EgAZIKECACAJuSoo7zzzp0jfNG++zrZgbVSr15e+8KFkjEard+qhdbr1FMjjKFIbr018b3qau983broYwGQGgkRAABNlP/zuUh9842TGN1ww6amC3Sf5qqnbhv/s0b3HV2ydVlXX+2dv/569LEASI2ECACAJuoXvyjwC6+5xkmMlizRqzpUE7SvdtU0JzMwxlmLtHRpWkMtXRpazK7kTZzonb/wQvHiAOAhIQIAoIn6zW+K9OJOndTstdd0rP6pvfSB1752rVeE4cYbkw7RubM0YUJhNq094wwnpFmz0n/mrrvC2/2b0U6dmltcAPKDstsAADQxpVLyOhDHW29JAwbEB9Wrl/PJnU/LltKGDd4YdXWFiXP77aUvv0zdT3Iq+P34Y/I+nTo5+9oCyD/KbgMAgITeeEM699xiRxHjwAOdzMbaYCW62bOdLKKsTJowQcce6yVDUmGTujlz0u+7dm3qPmvWZB9LqZg+3fmfZ8CAYkcCZI+ECACAJmbAAOnvfy92FEnMn+9kOpdf7rXV1Un9++uQf56n7pqrvn0LH5Y/EYv13nuZj5eXTXGLzP3nKJufHygVJEQAAKCo/IUG/Mwtf5WRU4RBLVvKSjpXD+pb9dKUaqPfa6QkGyglnm/33ZdevyefdI7+T+LctkSK/cliPsye7RxraooaBpATEiIAAFBU550X39aune+iUydp3Tpdd63V9pql8TpIkjRSwzVNfXTaKydI99yT1btXr05+/5pr0htn0iTnWFYmtW3rnF9wQbCPmzy4GkNC5G6s2xh+FjRdJEQAAKAoWrVyjjNmBNsXL5ZWrvSu3bU2Dz0kfaNtdUTz/0gvvaQaNdd6tdSJelG68EJneiaDzZV+/WsneenaNXGfZcvSG2vePOfYqpV0553O+apVwT5uclVWlnaIJe+HH5wjCREaMhIiAABQFPvt5xxj1+Z06RK8dmda3NmILbeUdOyxKrc12kNVOkBve50//9xJjMrLnfMkxo93josWpR9zomIJbgLXvr00ZEh4nzffdI6B2a8GLjbpAxoiEiIAAFAU//xnfNuoUd65O5Py8svOsbbWOQ4d6n+imf6nA3TKyVYaNsxrrq11ZouMkU47LedY3bVBf/xj+H03qdtmm8RjuLNNhSwIsX59cLYt39ati25soFBIiAAAQFFstll828UXO8eWLaWtt3bOV6wI9vEXn3M/d3v+eTkVEKyVvv3WmSFyPfWUk9F06LBpc6Dzzw+OecYZ8bG8+qp37n7eN2ZM+M+ycaNz7N8/2O6fUXILD1x5ZfgYUWjVKtoZqcZQKQ8gIQIAAEX3wAPO0d1k9fXXpT//2TmPXZ/Spo13/t//Bp+T5GRSGzY4D/qnY5Yvd7IwYzTu/tmSvJmfp56Kj8kt9tCihbTDDs55qo1UzzkneO2fUXJ/joMOSj5GFKZPj2Zcd9YOaMhIiAAAQNE0b+4c//CHYPsBB4TP2sTaaacUHaZMcTKRxx4LNH+tn+hD7aka20wD9Yra1y2Ne3T+fOf4q19JV1zhnKdKALbf3jluvrlzfPbZ5P0L9clZVBvxujNjQENGQgQAAIrmkEOc4+rV0nXXJe5X/6VbUrGfqwWccYaTGK1Zozmtf6K79Tt10jKVyWqcjtIidVWdMc6UUefO0mOPbZrRGTNGGjQo3Z/Isf/+ztGtwhbLnZl66aXMxs2WWxY83/wzc999F807gKiREAEAgKIZN847v+km5+iu1/GLnUHy69PHOU6YkMYL27TR9rVf6I+6Xbu2/koaN04jdLn+qj9phdo7fZYulc46S9O1k57QaWrdrb10662BYaZOlfbcM/Fr3E/w3KRq+fLgfbdgRKI1Sfn2/+3deXhU1fkH8O8bkpAQSMJOBCQogoIiYERxQRC1iFVEUVDqXrW4FbVW1Lq10rrij4p1qyIquNV9QxQVcWNTZAeFsAYBDYtAFpKc3x/vPd47a2ZCksnMfD/Pk+fcuducyXVo3r7nvKeuMlHe4YzTptXNexDVNQZERERE1CDY4WiPPx54bOpUbb21EqzZs6u/t63GLeIWN/j4YwCnnYaKu+/F7bgHLbANmDULyM8HUlKwFW1wLl7Rqg4334y12B+TcBHQrBnO67kQc+dqHYdgWrTwfX3HHdqmOH95NW6sbV3N7akv3oDom29i1w+ifSEmzlbSKigoMPPmzYt1N4iIiKiW2OFjlvdPE/9j+flAYWHoe2zdCrRqFXi8fXugqMh9nZ7uWyHNXm/fe+NGoEMHIBO7sOfgI4GVK7GoqjvysQbNsAsA8AO6IAMl6JhZjK4l3+EHdAvad2M061VWpvdcv97tT8uWmpAKZuhQ4Pe/DyzUEKmyMt9sW2mpG4jVFu/zOeEE4LPPavf+RDUlIvONMQWRnMsMEREREcWULawQTLNmvq+vvTb4efYP8549tZ082Xfe0aZN2nbvrgFKdeWi7Qi5UmkKLFsGVFbicFmE5tiGc/ESbsSDWIruaIOtQEkJVuJgbEUr7UhGBjB8OACdYDN3rvt+ixZpa9cr2rUr+PtnZwNvv71vxRD8gxNbNa+u2N8xUbxhQEREREQxNWqUu+0fHNmiC9YNNwS/h61It2mTDku7+GLfdY5s5ub558P3xa4bNH26tt4hek2bApVIxasYgfG4EWfiLTTBLqB/f4zBw1iFA/TEsjLgtdewG03xIkZiT9/j0R2LIKhArjNN6eijtbXD9+xwvvR0zeL8+qv7vnatpWjNnOn72i5wG6l27YDzz4/8fP95UkTxggERERERxZS3IvZtt4U+Fs6kSe52uNkAffqEv88DD2hrS27bAAYAevd2t21Gqgpp+GLcTEzAGByNOTou7bTTgLQ0bEMujsMXOAFfYAl6ogyZemFqKm7+eCBSUAljfKuz7d2rSygBbqnvzZtrVhRhyRJt7TC5aAOWzZuBF1+M/Pzdu6O7P1FDwYCIiIiIGgz/0tveLE9dspmpV1/V1g63697dPefBB91tb6Bgs04i0Ojj3XeB8nK8/Z8idMR6HIBVGI2JSIVTNaKyEm0XfYZf0BLzTW/8nHco+mI2clJ2ol07vc+bbwL//KdbhKFjR+C++7Q63dChkX2mDRu09R92GAkbEEbDBnJE8YYBEREREcXc6tX7XqXsoYeAzEzfTMV55wHjx1d/bU6OtmvXamsXHL30UvecI490t73rEs2Zo60tpW2NHg0AgkIcgMdxNcQYTV9dcw3QpAnexhlogy04FEswG0ejuCoXm34SVEkjDL2xC7BgAb7+Wu/188/A2LG67s/bb1f/eew1gM5HitY770R/DRdppXjFgIiIiIhirnNn4Kij9u0eN9ygmZ0mTdyhbq+9ppkWIHjJbuuww7T1XwD2ggt8X9uYxmvdOm2DrZ9k+QzVe+QRYPduXITn0AEbcRCW4zxMAeDcuKoKWLUK6N0bfY8STMdJGISPkI0Qq7yGYOchtWnjZpomTozs2miC09+GD1aFP4+ooWJARERERHEhPT3yc+36QHv3AsXFuu3N8Pi77jpto8ly2EBgmxOneOcb+fvyy9DHfkQ3vITzkWKjrUceAZo3/+34yZiBj3EKdqAFFqMHJuBaIC8PeOWVsP2z847y893CDHaOVHV++CGy84DA0uhE8YYBERERETVobdpoe8IJkV8zcqS7bTM64aqsDRsWfb/s3Bx7f1tK22vPHs2chMseBbjmGo3ibID0wgu//RIyUIrReFwrMYwY4Zana9kyYJVYu9Bt795Av366vXVrZF3wrtkUis2MpfCvSYpz/E+YiIiIGrQNG3TCvi2FXVPBFmzdFzbIsI49NvCczMzIMih2DlNQo0ZpyTdj0DVlFXKxHV83GeB7TnExcNVVboCUk4Oue3XRo4EDgUsu0dOqW3/J2rHD3bbzmPzNnq2t/9wponjDgIiIiIgatLS08PN/YsV/Ps7ll9f8XrbEdnVycoA9yMKIlp+6GaQ5c4ADDvBN1ezcicXoiWU4GAV9BaeNaIr2WB/xPB9vme933w1+ji3rHW5hXaJ4wICIiIiIEtJ++9XsuhkzIjuvS5fwr6vjzRzdfHNk1xxxhLabN3t2HnmkFmGorNQAaflyoEcPPIErsBadIACwezc2YH8UIU/fODNTV68NkTLyltD+9tvgfVm9Wlu7zhFRvGJARERERAnJWzo6mjk8Dz2kbV3PjanJULNx47TduzfMSd26AYsX4094AoPxIbBlC3DkkfgHbsNSOAsrlZYCkyfrL0ZEo5qhQ38LkLyV9AoLg7+NnWeUkcF5RBTf+J8vERERJSRvqevBgyO/7rPPtI2qEEIN2PtHE0z07autf+nvsFq3BubMwR24Bydhhk4QOvFE33GI5eW6wJETIP276moUYC5Ssfe3Kn3+7DpHzZq5n2Hhwij6RdRAMCAiIiKihDdlSvXn2LLeJSXatm1bd/0BgDPP1PbUU+v2ffyVNc7WcYHl5RpZlZZqdshT1/xq/Adz0Rc7kIO5Wztp1HPCCcAnn/x2ji280Ly5G1tFOtyQqCFhQEREREQJ68svdX5OkybVn+tZ+gcAcPzx1V8TzdpI/p5/XuORUEULapsdomfLj5eVAUOGQIfLvfmm7nAKNXTDUlyIZ/EkroDAALt2AZ9/DgwapEPs0tNxz7oL0BUr0K5VBTIz9Z7z5gF33aWnnH56/Xwuon0lJqqca+wVFBSYefPmxbobRERElGCGDAE++MB9vXAhcNhh4a/p3BlYs0a36+tPqpQUfa/Zs90hdMHYog22X1lZui7SGWdoUJSXp8sZ9e8PzJwZ/Fq7XbVshZbCmzlTV6L1fNhKpKAIeXgU16BT8x24t/wGrNvdGpmZ+n5EsSAi840xBZGcywwREREREYBbb/V9XV0wBABXXlk3fQnHlrm+7bborrPrMNl5Plu2aDtnTvjrjAHQrRt2Pfc6pPgXHNajCigpwdDUt3AZ/ovdKdlogWLci1swetu9WLW7Hb7BUXip5HQtizd1anQdJapnDIiIiIiIABx3XPTXjB2rbbt2tduXcOzcplDlsEPp0UPbrVu1tWsSedccCmfCBG2XLAGQkYEP5Aw8g8uw5IttuGzEHrTCVjzY6K94DFehFBn4HaZrJ0eN0jRTairQsSPw5z8D27dH13miOsSAiIiIiGgfGANs2lR/72eLMISLKTZs0NZbwe7cc7WNNADy9+qr2trRcpWV2h5xBFBQAPyCVrhF7sN1eAQDMBPZ2AFccolGiykpesGGDcC//60TtlJSdKXZk08Gvv66Zp0iqgUMiIiIiIjiyP33a2szPMF8/LG23rlA552nrQ1kouW/HpENjNLTgVNO0e2KCvd4OTKAZ57RaLGyUhdPmjgROPRQLeRgDLBzp3b2mGPc9ZC6d9fFoLw3I6pDDIiIiIiI4khubvXn2HlBdr4RoLGGNX687/kLFrjbc+dqm5LiG1Dt2uV7jbeIRM+evsdsRbs33/TsTE0Frr4aWLRI01TGAPPna8orN1ffrLwcWLYM+MtfUJLWFDslB2jTBvjDH9y0F1EtY0BERERElGBWrNDWGwR5Pfmk7+sbb3S3baW9Ro3cIXc//RQ+I+WvZUttn3iimhP79AHef18r11VVAb/+qsHQfgfgVZyDMqTrpKcpU3T+kQjQtCnQr59bP5xoHzEgIiIiInLYjIg3s9KQhZoPVFSkbVZW8ONr12qbna2td0WT777TNi3NXXD1jTd8r7eV6oIR0VFxgG/mKSJNmwIPPIAJ167CRXgebWWrTl7q29ddTGr3buCbb3RlWxHtZOfOuuBUTSdIUVJjQERERETksAutNmsW235Ux2ZuLrgg+PEdO7QNNbzOxg3PPqutdzicXVepSRM3BvniC9/rJ00K3bfmzYGLL9bt4uLQ5738sgZkGzcGHps1S1tjAAwfrosu7d6tOwoLgZEjgdat9RdRUaGdvv9+IDNT9zVvrivDhovciBwMiIiIiIgc772nbXVr88Rar17ahho1tnu3tnl5vvtT/P7yGzZMW+9wOLs+UU6OxhVA4O/jq69C9+3yy4ERI3R7797Q5/3xjzpCbsyYwGMrV4a+Dvn5wIsvakdtsYb77we6dtWI1hgtwffuu8Dhh2sWKTNTJzo9/jiLNVAABkREREREjkGD9O/pLl1i3ZPwZszQ1htwNG7sDvkrK9O2Wzff62wGLBybLWrfHujQQbdt1sgGVHbIXTB//7v7Pt7CC/7Jmj17tLVD9Lx+/jnwvJBSU4GbbtKJU2Vl+qazZunDzM7WX0ppqRZzGD1ah9ilpgL77adRmffNKCkxICIiIiKKM/5D4crKtEAboFkumwTp29f3vBYtQt/TLthqg6kePTTBArj3a99e2507A6+fNEmzPcGCrj599F4jR7r7bFbqp58Cz/cGQVOmhO5zSMcdp+W8d+zQN9q2TYOh9u21WkRlpZYDf/ppHXonouMkTzgB+OSTGrwhxTMGRERERERxyGaDbr8dGDLE3X/TTW5m5rTTfK85+GB3OyNDW1uJ7vrrtbXBz7HH6pqpXrfcoq0Nmrwuvhh4+OHA/pWXu1mgadMCryspCdznzXy9/37g8ajl5gL/+Y+W7q6o0F/QpElA797uL2LXLuDzzzWzJKKR3UEHAf/4B4fZJTgx3lxmHCgoKDDzvKVQiIiIiJLQ/vsD69dr4YOSEjcISktzAwr/P/MmTgSuvVa3u3UDli/XIGnFCk2UbNniBjI7dmis4C3dbYzv2kQioctxZ2Ro4HT77RpTADrkzi4M672Pfz+9x7p2dcuI793rVr6rdUuXAn/7GzBzpmaUgnWqRQtg6FDgjDM0yPL+NGsWOEmLYkZE5htjCiI6lwERERERUfyZOzdwSJw//z/zysrchMhDDwE33AA89hhw1VU6kqyiwg1G7LX+gYv3tTfA8dexoyZkUlJ8gyZjdFTaoEGh++l9j5wcrZEwaxbQv78ee/11rbrt9c03WqTBP6tVY6WlwD//CUydCqxbF75ChO10dnZgoJSbqx+iun3Z2fFT7z0OMCAiIiIiSgLewCE1NXBkV7A/8/wDHv990QREaWnu3CV/w4YBb74ZuN+YwGNFRb4V8UT0xxj3PU45BfjoI/ec/v01mWOlpOj5dfqn7ccf6xvZKG37dk2l2e1w+4JNvPLXrFnw4ClUUOXdn5MTWdWMJBFNQMQwlIiIiCgBXHut7xye2hZsNFijRuH74w16bIBTXq7ZLXvPqirg0UeBe+7RfZs2aZuerhktG+TNn6/tgQcCq1bpdB+vevn/+E86qebXVlZqCiuS4Mnu37gRWLLEfR1qfKLVpEnkAVSw/TZ9mGTqNCASkcEAJgBoBOC/xph7/Y5fDOABAHZJronGmP/WZZ+IiIiIEkWLFu7ip+PH6xyh6kZ2nXlm6ETCggWhr8vM1NY7BC5cQuLEE93tLl00C7Rnj45Cs5WuO3bUEt7TprkB0YsvapuV5VbRBjQmAIBly4CWLTW2eOYZ4NJL3QAL0KF1xx8ful8x06iRG3zUhDFa+CFU8BRs39atwA8/uPuqKw7RuHHNhvvZn8xM3xRinKizgEhEGgF4FMDJADYAmCsibxtjlvqd+rIx5pq66gcRERFRonrkEWDUKDdT06WLBgzhvPFG4L7sbB3Rddlloa+zJbcbN3Yrw0WaUJg1CxgwQIsj/O9/7jC7a68F/vIXYPVq91w7DK5tW9/aBjYIS0sDDjlEF4t9/HENiO67z71+0qQGGhDtK1savFkzjSSjZYxGpJEO8bM/a9Zou21b6PGR1nnn6ZyrOFOXGaK+AH40xqwGABF5CcBQAP4BERERERHVwPnnA82ba7ABAOPGAWedFf19Tj9d1/tZvDjwmM0IDR6sr7Oz3YDIZo1C6dJF/wZv1w44+2zNDhUWukHO6NEaEP36q3vNypXaduumyY2KCjfrZYftXXWVBkTLl+vrr792r589O/LPnVRENO2WlaWL0tZEaWn4gKpr19rtcz2ps6IKIjIcwGBjzB+d1xcAOMqbDXKGzP0LwFYAKwFcb4xZH+6+LKpAREREFFq4ctaheKvP+V/bvr0Od9u0SQObo47SYAQAevYEvv8+svfYtUuTG3YukX0P/yIOLVpoMmLiRODWWzVzddFFwOTJGozt2OF+Tlv2Oz3dDZpyc/X6WJk/X9eF+uYboHPn2PUj2UVTVKEui6UHG0Do/7V8B0C+MaYngI8BTA56I5ErRGSeiMzbapdRJiIiIqJa4V1ryN/GjRqstGunr485xj3WqlXk79G0qbbBqtd57d6t7YgRujYSoMPsAOCww3zPtffyzpvatSuy/ixZopmx2nb99bqe0x//WPv3prpRlwHRBgDeAY4dABR5TzDG/GKMsWsdPwXgiGA3MsY8aYwpMMYUtLbfDCIiIiIKYAONfVkjNNy1l1zibtsgqSZsQQY7/8kGNbZt1Qo46CDdtkGSXVTWe503YwSEXhfJ36mnAu++CzzwQPR9D+eHH7T1DuOjhq0uA6K5AA4Skc4ikg5gJIC3vSeIiKfiPM4AUM00QCIiIiIKJydH23AlsYOxGRwg/PqgPXu62wceGN17eAMt+/9xZ2Vpa+fie4f5+S+yOmKEu92ypbY2SLJZrkiHCf7yi7b/reX6xrbqX0mJG6xRw1ZnAZExpgLANQA+hAY6rxhjlojI30XkDOe060RkiYh8D+A6ABfXVX+IiIiIksEf/qBttNWdhwxxt8MNofPq0SO69/AGXQXO7A5bMC1YcbILLwx9r759tX35ZW3z8kIPwxszRo+99Za7r8wZo1RYWH2/o+EtxPa3v9Xuvalu1GWGCMaY940xXY0xBxpjxjn77jDGvO1s32KM6WGMOdwYM9AYs7wu+0NERESU6B55RMtQrw9bpirQc8+5297AJZwjgk52CO3gg91tm9mxFfK8Fe5sYOOdo+SftbrjDm1tADJokFsYwn89JRsIPfaYu88Oratu3aZobHRW1mzbVls79yla27dr5uzOO2unXxRenQZERERERFT//vrXyLM8lvf8Fi0iu6ZLl+jeY/Rod9su3Hr99doWF7sLtqalBV7r36cjj/R9ffvt7jn+w+Dsoq52fo+/Tz4J3+9QSkqA3r2Bdev09QsvaFtQoAHcTz/V7L5PP63lyr0ZLao7DIiIiIiIyEd+fvjjkS7I6u/iiwP32XlIpaXA//2fbtt5RV7e6nbB7L+/m4H68kvfY3v2aGsDrooK3+N33x3+3qHcdJNmo845R19//LG2v/udLh4L1GydUhug2WISNZWfD4wcuW/3SAYMiIiIiIgIgC6GCvhWcwumpCTy4gXB+A9/s0Pvxo3Ttk0b95gtDhFs+JitVGfZeVBr1/rutwGQDYxs4GIzSt9+G1m//dk1mBYu1NYuFDtqlGbpAOD++6O/79Kl2noXrPX6+WdgWTWlyIqL9ffAnYUZkgAAFD9JREFULFP1GBAREREREQD9g94YzXDUFWMC5+3Mm+dbgc6W2waAe+7RzE+vXoH3at9eWxsYXXaZtv5rEVVVaWvnDU2bpq3NhEW6dpE/G3iVluo9bAaqRQu3uEWowOXyy4HMTA0u/W3Zou3OncGv7dUL6N5drz/uOHdIoNeHH7p9swEbBceAiIiIiIhizmZtAGDgQHd77NjQQYUNgOxQOVty3H9InGWzWrbowiGHuMP/bDATjW3b3O1x4zT48M7FatpUiz74r41UWanzhEpLgVdfDbyvDZJKS4O/7+bN+j6NGunwwKefDjxn5kx321tMggIxICIiIiKimBs4EBg8WCvM/elPkV1z223AG2+4Q9cs73A+/+Bo1y43szNwoGZaAHeIWzRKStyKeLaggl1fCXAzXe+953vd5Ze7fXznHd9j5eXuMWMCsz/FxfqZ+vUDpkzRff5DBAF3GKAIMH165J8pGTEgIiIiIqIG4YMPdHhbkyaRX3Pmmb6v/dci+vxz39evv+4unjp0KHDjje57R6uyUrNAKSnAhg26z87DAoDTTtP2mWd8r3v+eXfbv0S4DZBs5mr+fN/jn36q7WGHuVX+iooC+7Zmjd6jQwfdtsMGKRADIiIiIiJKGHbI2qpV2s6Yoa0t5DB9ujskrVUr4PzzdXvr1ujex57fooU7lwnQ9ZCs667Tdu5cd9/dd2uGxw7z27TJ9742ILKBlX/266uvtO3Xz81A2TlHXsXFQLt2Wt68qorFFcJhQERERERECSM3V1s7b8YWFGjXTttFiwKH0TVqFDjPpzo2o9SpE3DWWe7+UaPc7datNXvkDVjuu0/b11/X9ZZs5TvLDnWzgZqtXGfZzzNokBaTEAF++cX3nHXr9PN06wZceaXumzw5us+XTBgQEREREVHC6NpVW7uWT2Ghtqeeqm1Rkc7N8Va1s+W333038veZNUvbnj191zHaf3/f83JyNAArL9f7l5QAeXla0CE3V/tSXu6eb4feXXKJtmvW+N5v9WoN4Gxp8rQ0YMcO33PsnKW+fTWTlJ4euDYTuRgQEREREVHCuPRSbe2QOTu07fLLtbVr+6Sludf076/tgw8G3m/0aA1qRo3yrfq2eLG2J56ox/PyfIfOWT16aDt5MnDVVbo9YYK2dg7QG2+45+/cqcFa69aa/dm40fd+mzcD2dnu68zMwLLhX3yh7SmnaNutm1bRq2l58UTHgIiIiIiIEoYdamb/+Ldt797a2mxMVpZ7zR13aOtf4AAAnnpKg5SpUzX4uOkm3b9+vbYnn6xtUZGb3fE6+2xtn3tOr0lPB845R/cdd5y23vk9tlADoEURvOXAq6p0iN1++7n7mjYNXMto0SJtjzlG2759tZ03L7B/xICIiIiIiBKIzfzYqmplZdqmpmrmxZa09pbH7tlT22ALoVZWaqbmsMP09cSJ2to1iGzwEoqdw2OzNsOGucds8Pbdd9ragMwGPNnZvn1atkz7b4cFApqd8l/odv16t/od4J5vAyXyxYCIiIiIiBKKDQT27vUtN52e7m7bCm1WWpoGG96CCzbjk5WlxQxE3GFzpaVu5bpwMjN1zo81aZK73auXtnZY3Msva2sDtNat3YAOcBdb7dPH3deype88pKoqnVPkHb536KHa+hdo8Pfhh27Rh2TCgIiIiIiIEorN2jz/vG8BhWbN3HPsMDLLZmVsBggAHn5Y2wMO0DYvT9sHHohuvaRWrbTt0kUDJK/UVGD3bt22WSRbAKJ9e+2/rVI3Z462ds4T4BZX+OEHbRcv1mvsgrOAG3jZAhOhPPmkFojwLmybDBgQEREREVFCsWv4PP20tnYYXceO7jlDh/pec/rp2nozOB9+qO3gwdqOGaPtQw9pawOd6owYoe1TTwUey83V4Kqy0l2EdfhwbTt31tbuX7pU26OPdq+3gdyPP2o7bZq2xx4beI5/gQZ/hYU6H8k7bykZMCAiIiIiooRy0UXa2jkzNpNj5wEB7rA06847tbWZFgBYu1bb66/X1hZU2LxZ2/z8yPozYYJmXQYMCDxmg57OnTUY6djRzXDZxVvt51i/Xhee9Q7969RJW5v9sYUTBg70fZ/GjasPdOw9/Et9JzoGRERERESUUK64QltbYtuuM/T734e+xmZ7vBXb7FA2u6groJXfLFu5bl/YSnDr12sma8UK99jhh2tr5/4UF+ucIS8blK1bp60tN26HyVlNmwauV+S1fbv+AG4gCAB33eWWLA/n0Ufd7FS8YUBERERERAnFu8YQ4BYYsAGRSPDr7PweG5QY41sQAQCOOsrdPvHEfesn4FaaA3QOkXeOkS2esGaNFnEoL3czQpYtDlFU5LaNG/suPAtoUOhfntvLO7/ImyF66y3fsuCADvHzFp/Yswe4+WbgtddC378hY0BERERERAnHG/TYxVFtdidUqWy7ptDw4cDcubqdk+N7zhNPuNsnnbTv/ezbVwOfRx4JLPSQna2BzaZNwC236D77WSy7uKsdxrdtG9C8eeD75OW56xgF4w2IbIaoqgpYuVIXt7XZNgA46yx3fSUAeOcdzaaNGhX+szZUERQLJCIiIiKKL1lZ7qKsdgFUIHwFtZdf1gzNkiU67wfwrdYGaMEGEb2Pdy7PvrBFE4LJyNCM1fLl+pluv933eJMm2p9fftEApqzMd+FWyxaUWLTIN8tl2YCoY0c3INq40Q2gVq92h/B99ZUGSYsXa0nvqVM1C3f88ZF/5oaEGSIiIiIiSji2VDYADBkS2TUZGZqVMUazHgBw7rmB540fD4wdWz/lqXNy9H3at9ehbPvvH3hOaqrO/7ELux54YOA5dmjd998Hf5/CQn2v3r3dIXMrV7rHV6/Wdts2DYYADRqLi4EPPgBGjgwcXhgvGBARERERUcLxBjK5uZFfd8cd2u7cqW2wggJjxgD/+lfouUi1adw44IILNEgJVeY7I0OHtH31lb72r6AHuEPtQi3OWliole46ddIMkTG+BR5ssQZbhS8/X9d5euwxXQDXOxcq3jAgIiIiIqKEY0tlR+vGG31fe6vKxcIllwDPPadZoFCaNtWCCTZDFGxInK2IF2pxVhsQ5edrMLh9uwZEWVk6J8lmiGyQNH68Ds+7804dRlgbFfdihQERERERESUcu/ZQTdjhZf7V6hqqnBzN0tghbv36BZ5j1ztavz7wmDGagbIZIkCzRCtWAF276hA8myFauVKHxp12mi5YW1mpxRTqI1tWVxgQEREREVFCmjhRCyVE66WXtD300NrtT11p2VILKqxdq5mkYFX0UlI0wNuyJfDY5s2aYbIZIkADpBUrNPtz4IG+GaLOnbWgxK23Ah066JC+eMYqc0RERESUkK6+umbX9elTPwUTakubNtpu2gQ0axb6vKwsd/FVLzuMzpshWr5cA6yLLtL1j157TdceWrlSs0aAVpULlnGKN8wQERERERHFsbw8bffuBdq2DX1e8+bB1yHyBkQtW2rgNGOGBoU2Q1RRAaxbp0UVunWr/c8QSwyIiIiIiIjimF1jCHDnCgXTtq3O+amo8N1vA6L8fJ0L1KkTMGuW7uvWzS1hPmuWBlQ2Q5QoGBAREREREcUxbxB0yCGhz7OBk3/p7cJCDZZsIYpOnbSCHKAFJuy6Rh98oC0DIiIiIiIiajC8C7EWFIQ+r0sXbW15bsuW3LZsYYX99tM5Se3ba0GG6dN1P4fMERERERFRg+HN2Bx/fOjzunfXdulS3/225LZlCyvYwKdRIz2+bZvOL9pvv33ucoPCgIiIiIiIKI5lZ2sromWwQ+nVS9uFC7WtqgI++USLJQTLEHkzQXYeUdeu8b3mUDAsu01EREREFOdSU4HGjcOfc/DB2r73nmZ9MjK0SEJuLjBsmHuef4YI8A2IEg0zREREREREcW7AAOCMM8Kfk5oKvP8+cPbZWmAhPR144QWgqMh37lGvXsCFFwJDh7r77DylRAyImCEiIiIiIopzH30U2Xmnnqo/4WRkAJMn++6zGaJEK6gAMENERERERETV6N8fGD4cOOmkWPek9jFDREREREREYbVoAbz6aqx7UTeYISIiIiIioqTFgIiIiIiIiJIWAyIiIiIiIkpaDIiIiIiIiChpMSAiIiIiIqKkxYCIiIiIiIiSFgMiIiIiIiJKWgyIiIiIiIgoaTEgIiIiIiKipMWAiIiIiIiIkhYDIiIiIiIiSloMiIiIiIiIKGkxICIiIiIioqTFgIiIiIiIiJIWAyIiIiIiIkpaDIiIiIiIiChpMSAiIiIiIqKkxYCIiIiIiIiSlhhjYt2HqIjIVgBrY90Pj1YAfo51J6he8FknBz7n5MFnnTz4rJMHn3XyqO5ZdzLGtI7kRnEXEDU0IjLPGFMQ635Q3eOzTg58zsmDzzp58FknDz7r5FGbz5pD5oiIiIiIKGkxICIiIiIioqTFgGjfPRnrDlC94bNODnzOyYPPOnnwWScPPuvkUWvPmnOIiIiIiIgoaTFDRERERERESYsBERERERERJS0GRDUkIoNFZIWI/CgiY2PdH6pdIrJGRBaJyAIRmefsayEiH4nID07bPNb9pOiJyDMiskVEFnv2BX22ov7tfM8Xikif2PWcohXiWd8lIhud7/YCERniOXaL86xXiMjvYtNripaIdBSRT0VkmYgsEZE/O/v5vU4wYZ41v9cJRkQyRGSOiHzvPOu7nf2dRWS2871+WUTSnf2Nndc/Osfzo3k/BkQ1ICKNADwK4FQA3QGcJyLdY9srqgMDjTG9PDXuxwKYYYw5CMAM5zXFn2cBDPbbF+rZngrgIOfnCgCP1VMfqXY8i8BnDQAPO9/tXsaY9wHA+Td8JIAezjX/cf6tp4avAsCNxphDABwN4GrnefJ7nXhCPWuA3+tEUwbgRGPM4QB6ARgsIkcDuA/6rA8CsA3AZc75lwHYZozpAuBh57yIMSCqmb4AfjTGrDbGlAN4CcDQGPeJ6t5QAJOd7ckAzoxhX6iGjDGfAyj22x3q2Q4F8JxR3wDIFZG8+ukp7asQzzqUoQBeMsaUGWMKAfwI/beeGjhjzCZjzLfO9q8AlgFoD36vE06YZx0Kv9dxyvl+7nJepjk/BsCJAP7n7Pf/Xtvv+/8ADBIRifT9GBDVTHsA6z2vNyD8F5LijwEwXUTmi8gVzr62xphNgP6jDKBNzHpHtS3Us+V3PTFd4wyVesYz9JXPOgE4w2R6A5gNfq8Tmt+zBvi9Tjgi0khEFgDYAuAjAKsAbDfGVDineJ/nb8/aOb4DQMtI34sBUc0EizhZvzyxHGuM6QMdWnG1iPSPdYcoJvhdTzyPATgQOgRjE4CHnP181nFORJoCeA3AGGPMznCnBtnHZx1Hgjxrfq8TkDGm0hjTC0AHaGbvkGCnOe0+PWsGRDWzAUBHz+sOAIpi1BeqA8aYIqfdAuAN6Bdxsx1W4bRbYtdDqmWhni2/6wnGGLPZ+R/ZKgBPwR0+w2cdx0QkDfoH8hRjzOvObn6vE1CwZ83vdWIzxmwH8Bl03liuiKQ6h7zP87dn7RzPQeRDphkQ1dBcAAc5lS7SoRP23o5xn6iWiEiWiDSz2wBOAbAY+owvck67CMBbsekh1YFQz/ZtABc6VamOBrDDDsGh+OQ3V2QY9LsN6LMe6VQq6gydcD+nvvtH0XPmCTwNYJkxZrznEL/XCSbUs+b3OvGISGsRyXW2MwGcBJ0z9imA4c5p/t9r+30fDuATY0zEGaLU6k8hf8aYChG5BsCHABoBeMYYsyTG3aLa0xbAG85cvFQAU40x00RkLoBXROQyAOsAnBPDPlINiciLAAYAaCUiGwDcCeBeBH+27wMYAp2IuwfAJfXeYaqxEM96gIj0gg6lWAPgSgAwxiwRkVcALIVWsrraGFMZi35T1I4FcAGARc58AwC4FfxeJ6JQz/o8fq8TTh6AyU5VwBQArxhj3hWRpQBeEpF7AHwHDZDhtM+LyI/QzNDIaN5MogieiIiIiIiIEgqHzBERERERUdJiQEREREREREmLARERERERESUtBkRERERERJS0GBAREREREVHSYkBERET1QkR2OW2+iJxfy/e+1e/1V7V5fyIiSlwMiIiIqL7lA4gqIHLWogjHJyAyxhwTZZ+IiChJMSAiIqL6di+A40VkgYhcLyKNROQBEZkrIgtF5EoAEJEBIvKpiEwFsMjZ96aIzBeRJSJyhbPvXgCZzv2mOPtsNkqcey8WkUUiMsJz789E5H8islxEpoizGrOI3CsiS52+PFjvvx0iIqpXqbHuABERJZ2xAP5ijPk9ADiBzQ5jzJEi0hjAlyIy3Tm3L4BDjTGFzutLjTHFIpIJYK6IvGaMGSsi1xhjegV5r7MA9AJwOIBWzjWfO8d6A+gBoAjAlwCOdVZBHwbgYGOMEZHcWv/0RETUoDBDREREsXYKgAtFZAGA2QBaAjjIOTbHEwwBwHUi8j2AbwB09JwXynEAXjTGVBpjNgOYCeBIz703GGOqACyADuXbCaAUwH9F5CwAe/b50xERUYPGgIiIiGJNAFxrjOnl/HQ2xtgM0e7fThIZAOAkAP2MMYcD+A5ARgT3DqXMs10JINUYUwHNSr0G4EwA06L6JEREFHcYEBERUX37FUAzz+sPAYwWkTQAEJGuIpIV5LocANuMMXtE5GAAR3uO7bXX+/kcwAhnnlJrAP0BzAnVMRFpCiDHGPM+gDHQ4XZERJTAOIeIiIjq20IAFc7Qt2cBTIAOV/vWKWywFZqd8TcNwJ9EZCGAFdBhc9aTABaKyLfGmFGe/W8A6AfgewAGwF+NMT85AVUwzQC8JSIZ0OzS9TX7iEREFC/EGBPrPhAREREREcUEh8wREREREVHSYkBERERERERJiwERERERERElLQZERERERESUtBgQERERERFR0mJARERERERESYsBERERERERJa3/B2rkv2/yEb0yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "hidden_size = 128\n",
    "model = QANetwork(hidden_size)\n",
    "optimiser = optim.Adam(model.parameters(), lr=0.0005)\n",
    "train_losses, test_losses, test_acc = [], [], 0\n",
    "epochs, iters_per_epoch = 10, len(train_data)\n",
    "\n",
    "plt.figure(figsize=(14, 8))\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Loss')\n",
    "plotted_legend = False\n",
    "\n",
    "\n",
    "def plot():\n",
    "    global plotted_legend\n",
    "    plt.plot(range(len(train_losses)), train_losses, 'b-', label='Train')\n",
    "    plt.plot([(i + 1) * iters_per_epoch - 1 for i in range(len(test_losses))], test_losses, 'r-', label='Test')\n",
    "    clear_output(wait=True)\n",
    "    display(plt.gcf())\n",
    "    if not plotted_legend:\n",
    "        plt.legend(loc='upper right')\n",
    "        plotted_legend = True\n",
    "\n",
    "\n",
    "def train():\n",
    "    model.train()\n",
    "    for i, x in enumerate(train_data):\n",
    "        optimiser.zero_grad()\n",
    "        y_hat, _ = model(x)\n",
    "        loss = F.cross_entropy(y_hat, x.answer.squeeze(1))\n",
    "        loss.backward()\n",
    "        train_losses.append(loss.item())\n",
    "        optimiser.step()\n",
    "        if i % 10 == 0:\n",
    "            plot()\n",
    "\n",
    "\n",
    "def test():\n",
    "    model.eval()\n",
    "    test_loss, correct = 0, 0\n",
    "    with torch.no_grad():\n",
    "        for x in test_data:\n",
    "            y_hat, _ = model(x)\n",
    "            test_loss += F.cross_entropy(y_hat, x.answer.squeeze(1), reduction='sum').item()\n",
    "            pred = y_hat.argmax(1, keepdim=True)\n",
    "            correct += pred.eq(x.answer).sum().item()\n",
    "\n",
    "    test_losses.append(test_loss / len(test_data.dataset))\n",
    "    return correct / len(test_data.dataset)\n",
    "\n",
    "\n",
    "for _ in range(epochs):\n",
    "    train()\n",
    "    test_acc = test()\n",
    "plot()\n",
    "clear_output(wait=True)\n",
    "display('Final test accuracy: %.2f%%' % (test_acc * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARoAAAEGCAYAAAC6p1paAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnXm8XdPd/9+fJAiCIGgkSB9Ta4iomKdow5O2ij6lppaUUlXVRyuqE0FrylOe1lDUL6giMZTSehpKEjGExJRIBEFUzKGGFKnkfn9/rHVyd87d5+x9cs8599xzvu/72q+795r3cL57rbXXWh+ZGY7jOLWkR1cXwHGc5scNjeM4NccNjeM4NccNjeM4NccNjeM4NccNjeM4NccNjeM4NccNjYOkSZK+3UV5/1TSlV2Rd14kmaRNurocnUGBqyT9U9Ij0e27kt6QtFDS2vH/f9Qifzc0DYikkZJmSvpQ0uuSLpW0Rs64O0t6X1LPhNvvS7hdVovyV4KZnW1mFRs5SbtJelDSe5LekfSApO1rUcbOIGkbSXfHMr4saWSOOJOiQVipikXZDdgbGGhmO0haAbgA2MfM+pjZ2/H/C1XMcyluaBoMST8CzgNGAWsAOwGDgLviw5HFdKAn8LmE2+7Aq0VuewD3VaHIS4lvzZo/U5JWB/4CXASsBQwAzgAW1TrvlLL0zAiyAXAJ0B84GLhc0vpl0htEuF8G7FedUgKwETDPzP4Vj9cDegOzqphHaczMtwbZgNWBhcDXi9z7AG8CR+ZM5x7gR3F/XeAFwg8x6WaEtxvAJOAs4AHgA+AuoF8ivZ2AB4F3gSeBYQm/ScCvYtyPgE0IBvL/Aa8BrwC/BHqWKOto4I9xvzfwR+DtmNc0YL2UOEOBd8uc/8bAvTGdBcB1QN+E/zzgZGAG8B4wHuid8B8Vy/4qcFS8VptEv6uB3wF3Av8ChgNfBh4H3gdeBkaXKFeveH+3LlP20+K1vAD4S4XPz/rA7cA7wFzgmOh+NPAxsCTmf0Msu8Xje2O45HmuDPwaeCleo/uBlbOeh5Jl6+ofl2/LPCgjgMVArxS/a4Dr4v5uGT+004E/x/0DgT8Qqs1JtxcS4ScBzwObxQdsEnBu9BsQf7BfItSA947H6yTi/gPYMv6QVgBuAy4HViUYtUeA75Qo62jaDc13gDuAVQi1su2A1VPirB7LcA3wRWDNIv9NYjlXAtYh1Nz+N+E/L5ZpfUKN6GnguMQ9eAPYKpb/ejoamveAXeP16A0MA7aOx4Nj/ANSyv3bmG+PMvduLnB8PPdPSBha4DBgRpm4k4FLY5mGAG8BX4h+I4H7E2EHxfPqlXBLnucl8d4OiPdil3g9yz4PJcvW1T8u35Z5UL4BvF7C71zgrpzpDIs3X8BvgGMItaI3Em5XJcJPAn6eOD4e+Fvc/zFwbVH6E4i1qxj3zITfeoQmzMoJt0OBiSXKOpp2Q3MU4U05OMc5fjb+6OcTjPPtpNR+YtgDgMcTx/OAbySOzwcui/tjiUY2Hm9GR0Pzh4yy/S9wYZHbKcAzwKfKxNstGpd+8XgOcFLOe74BocayWsLtHODquJ/b0EQD8hGwTUo+ZZ+HUpv30TQWC4B+knql+PUnvKHyMJVgWLYi9MVMMbOFhGp9wa24f+b1xP6HMT6Etv1Bkt4tbIQfRP9E+JcT+xsRajWvJcJfTqjZZHEt4aEdJ+lVSeeX6pcys6fNbKSZDYzntD7hB46kdSWNk/SKpPcJzbF+Oc93/aLzeSkl+6Q/knaUNFHSW5LeA45Lye+/CU2Z1ynNkYSXyYJ4fH10y8P6wDtm9kFR2QfkjJ+kH6FW9HyKX57noQNuaBqLhwi1gf9KOkpaldBEmJwnETP7mNC/sS/Q38zmRK8p0W0w+TuCXya8wfomtlXN7NxklkXhFxHeyoXwq5vZljnK/YmZnWFmWxCq6vsCR+SIN4dQ09gqOp0TyzTYzFYn1BSVfapA6JvZIHG8YVqWRcfXE2pUG5jZGsBlKfn1J/T5pCJpZeDrwJ7xS+PrwEnANpK2yVHuV4G1JK1WVPZXcsQtZgGhT2fjFL88z0MH3NA0EGb2HqHT9iJJIyStEL9C3ER7p2Ze7iO8RR9MuN0f3V43s7S3VRp/BL4i6T8l9ZTUW9IwSQNLnMNrhM7kX0taXVIPSRtL2jMrI0l7Sdo6fsl5n9CMWJIS7jOSflQog6QNCM2zqTHIaoROznclDSB07ublRmCkpC0krULo78piNUJt4mNJOxD6UorpD7xYJo0DCOe6BaF/ZQiheTiFfMb2ZcK9Pifeo8GETuBKnplCWm2EJuQFktaP933n+Lm9ouehgBuaBsPMzgd+CvwP4QvQi4TO0eEWP01K2l3SwoykJhOaK/cn3O6Pbrk/a8cHeP9YprcIb7RRlH92jgBWBGYD/wRuJqNqHflUDPs+oYN2MuHBLuYDYEfgYUn/IhiYp4AfRf8zCJ/y3wP+CvwpR94AmNn/EZpg9xI6Zu/NEe144ExJHxC+Gt2YEmYuodlRiiMJ/Wb/MLPXCxtwMXC4pF6SDpdU7nP0oYS+l1eBW4HTzezuHOVP42RgJqFm/A5hyEWP5XweUOzMcRoUSUcRfji7mtk/uro8jrM8uKHpBkj6JvCJmY3r6rI4zvLghsZxnJrjfTSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49QcNzSO49SctLVpHafhkVTXZQfMLO9SoKmMGDHCFixYkBnu0UcfnWBmIzqTVyPihsZx6sCCBQuYNm1aZrgePXoUL2reFLihcZw60dbCaz+5oXGcOmBAKy8y54bGceqCYR1UWloHNzSOUw8MlrS5oXEcp4YY3kfjOE4d8D4ax3Fqjhsax3Fqipm1dNOp4ikIkn4maZakGZKekLRjNQoiaZKkoZ2If4CkLapRlox8Rkq6OO6PlnRyhfE7SNlK6ivp+ArTWSZO1D/+SyVpFKV3taQDU9yvLFxXST9d3nScYGyytmalIkMjaWdgX+BzZjYYGE7Q3q0ZUfA9DwcQBNLT0mj0mltfgn5zreNUjJl928xmx8NMQ+OkY8ASs8ytWam0RtMfWGBmiwDMbIGZvQog6TRJ0yQ9JekKSYrukySdJ+kRSc9K2j26ryxpXKwZjQdWLmQiaaGkMyU9DOxcKu1E+F2A/YAxsZa1ccz3bEmTgR9I+oqkhyU9LunvktaT1EPSPEl9E2nNjX7rSLol5jtN0q7lLkzM82+SHpU0RdJnovunJT0U0zirRPRzgY1j2ccoMCae70xJB2fFiW59JN0saY6k6xL3YDtJk2PZJkjqn3EuZ8WaSY9CTVPSucDKMb/rYrgj4v17UtK1iST2kPSgpBeStRtJo+J1mCHpjOg2SNLTkn6vUFO+S9LKNCGtXKPJdfKJi9AHeAJ4FrgU2DPht1Zi/1rgK3F/EvDruP8l4O9x/4fA2Lg/GFgMDI3HBnw9K+2isl0NHJg4ngRcmjhek3YJ4G8nyvQb4Ftxf8dE+a4Hdov7GwJPx/2RwMVxfzRwcty/B9g0kc69cf924Ii4/z1gYUrZBwFPJY6/BtwN9ATWA/4B9M+IMwx4DxhIeIE8BOwGrAA8CKwTwx1cuO5p1w84H7g8ca0mJe7LwkT4LYFngH7JexTTuSmWYQtgbnTfB7gCUPT7C7BHPI/FwJAY7kbgGyWev2OB6XGzem6V/E7Stm223dbefP/9zA2Y3tm8GnGrqElhZgslbQfsDuwFjJd0qpldDewl6RRgFWAtYBZwR4z6p/j/0fhgER+y38Z0Z0iakchqCXBL4rhc2uUYn9gfGMvbH1gReDER5jTgKuCQRJzhwBaJytPqklZLy0RSH2AX4KZE+JXi/10JhgOCkTwvR7l3A24wsyXAG7FWtj3BaJXjETObH8v0BOFavwtsBdwdy9YTeK1E/F8AD5vZsTnK+HngZjNbAGBm7yT8bjOzNmC2pPWi2z5xezwe9wE2JRjRF83sieiefEaWwcyuIBirus/e7jTtxrIlqbjvIj78k4BJkmYCR0oaR6jhDDWzlyWNBnonoi2K/5cU5Vnqyn8c80FS74y0y/GvxP5FwAVmdrukYYTaCIQ3/yaS1iH08/wyuvcAdjazj5IJFrXaSIR918yGlChHpU/Y8i5JsCixX7jWAmaZ2c454k8DtpO0VpHhKFXGUue1qChc4f85Znb5MolIg1LK3XRNp2rOdZI0glAT7wlcaWbnFvlfSKgIQHg5r2tmfelCKu0M3lzSpgmnIcBLtP/wF8S3e56vDvcBh8d0tyI0n9LIm/YHQGqNI7IG8ErcP7LgaOHu3wpcQGgevR297gJOKISTVMqIYGbvAy9KOiiGlaRtovcDhJoSxPPNUfb7gIMl9YwGcA/gkYw4pXgGWEehIx9JK0jaskTYvxH6fv5aovb2iaQV4v49wNclrR3TXSujHBOAo+I9RNIASevmKH/TsKStLXPLQuHjyCXAFwlN00NV9LXVzE4ysyHxxXcR7S2KLqPSzuA+wDWSZsemzhbAaDN7F/g9MBO4jfBmzOJ3hM7LGcApdPwhAVBB2uOAUQqdvRun+I8mNG2mAMUrEI0HvsGyTa0TgaGx43I2cFzG+RwOHC3pSULTbv/o/gPge5KmEYxdB6JxeyB2/o4hGL4ZwJPAvcApZvZ6RpxUzOzfBON8XizbE4RmXqnwNxGu9+0pnbJXADMkXWdms4BfAZNjuheUSjOmexeh3+uhWBO+mXyGskmwXH852IHQ7/VCvLfjaH/W0jgUuKEKJ9Ap1MrtRqf7Uu8+GuvkCnuDhwyxv957b2a4Ddde+yWWfRFeEfumAIhf8UaY2bfj8TeBHc3sBIqQtBEwFRhY6IroKhp9fInjNA05X+oLzKzcwNU0g1cq4UMIHfZdamTADY3j1I0qtR7mAxskjgcCr5YIewhhSEWX44bGcepAFZeJmAZsKunThI8bhwCHFQeStDlh7NhD1ci0s7ihcZx6YJbrq1J2MrZY0gmEr3g9CYMvZ0k6kzDYrzDW6lBgnDVIJ6wbGsepE9X6zZvZncCdRW6nFR2PrkpmVcINjePUgTCXoSEqF12CGxrHqRMtvGSwGxrHqRcN0l3SJbihcZw64YbGcZyaYlX66tRdcUPjOHXCazSO49QU13VyHKcu+Odtx3Fqjn/edhynppgZbd4Z7DhOrfE+Gsdxao5/dXIcp+a0sqGpdHFyl8NtITncjDgL4/9Bkp5a3rxbBbOgvZ21NSu5DY1cDrdWNKwcrlNdqrQ4ebekkhqNy+GWQM0hh9tBxlZSH0n3SHoslqXcavtIulPS4Lj/uKTT4v5Zkr5dKr3o/4NEOr+SdGK5vLobBixps8ytackraYnL4UJzy+Gmydj2AlaP+/2AuYnruLC4LMCp8TxXJyw5OSG6TwQ2L5VeTOOx6N4DeB5YO+N57FaSuJttuaVNnD07c6PVJXHN5XCbXQ43TcZWwNmS9gDagAEE4/d6iTSmEPSwXgT+CuwtaRVgkJk9oyA+1yE9M5sn6W1J28b0H7d2Ib+lSDqWoL/dLWnmPpgsKtXedjncjjSLHG6ajO3hwDrAdmb2iaR5lL/+04ChwAuEWlk/4BjCSyYrvSsJNcZPAWPTEjfX3u62VNIZ7HK4KVjzyOGmsQbwZjQKewEblQtsQTnxZeDrBOGyKcDJ8X9WercCIwi1twkVlLFbUNDeztryIGmEpGdin+KpJcJ8XUFRdpak66t5LstDJZ3BLodbmqaQw03hOsJ1mB7PcU6OOFOAN8zsw7g/kHZDUzK9WNaJwI3WAIJntaAan7eVQ3s7Vgh+AuxqZlsC/139s6kMl8R1GgJJPYDHgIPM7Lkc4buVJO5mW25pv70hWwL7i9ts86iVUaqMtdPRZvaf8fgnsXznJMKcDzxrZld2pszVpKIBe45TC+IbeS5wTx4j0x0prEdThQF7A1h2/Nr86JZkM2AzSQ9ImippRHXOYvlp9MFsTgtgZrOB/+jqctSU/H0w/WLTssAVsRO8QB7t7V7ApoRhDwOBKZK2il0RXYIbGsepEzlrLAvKNZ3Ip709H5hqZp8QPlQ8QzA8efpPa4I3nRynDlTxq9NS7W1JKxK+ahaPsbqNMNYNSf0ITakXqnc2leM1GsepE3XU3p4A7BO/mC4BRqUNgKwnbmgcpy5Ub9KkZWhvx/FhP4xbQ+CGxnHqgFnYWhU3NI5TJ3yuk+M4NaeVB8e6oXGcOuACco7j1B5zuRXHceqB12gcx6k11sxLdWbghsZx6kQLV2jc0DhOPQjjaFrX0rihcZw64YbGcZwaY7Qt8a9OjuPUkFZvOnXpMhGSTNK1ieNekt5SJ+Rd64nKSPGW88ubnjohFazlkKpVkPxdP3E8Ly4z4FSBai1O3h3p6vVo/gVsJamgVLk37WoFuVDXSt6WlOLN8Fue9OrBSGD9rEBJuvj6dy8KMyvLbU1KVxsagP8Dvhz3DwWWruAsaQcFmdbH4//No/tISTdJugO4S9K1Ssi1KkjC7pfMRNKlBTdJt0oaG/ePlvTLuP8NBfneJyRdHlecL8j0/krSk3EN1vWUIsWbyCtNpndIjDsj5r9mUflKpXeQOkoK91SQzZ0W0/tOiWvbS9I1MczNCmJuqRLGCjK4Q4HrYv4F4/99tUvYFqR+R8d4dwF/kNRb0lUxzOMKUiqUcR8p6TZJd0h6UdIJkn4Yw0yVtFaph6U708J2piEMzTjgEAWxuMHAwwm/OcAeZrYtQVHy7ITfzsCRZvZ5gvjYtwAkrUGQFFlmvQ6CXtLucX8A7TWH3Qhrqn6WIBm7axSDW0K7FtOqhKURt4npHGNmDxJWNhtlZkPM7PlCRiX8/gD82MwGE+RjTk8Wrkx6vcxsB4JkRiHO0cB7ZrY9QQfpGEmfTrm2mxPWnB0MvA8cH90vNrPtzWwrgu75vmZ2MzAdODzmXxDPW2BmnyNI5JycSHs7YH8zO4wgg4uZbU14WVwT72cpdwgKmocBOwC/Aj6M9/kh4IiUc+neWOgMztqalS43NGY2gyDfeigdjcMaBD2mp4ALgaT42d1m9k5MYzJBcXLdmM4tZra4KK0pwO6xD2Q2QW62P8FgPQh8gfDjmaYgKfsF2hfM/jdQ6DdKSvvmIhq/vrGcANcQhOHykCYpvA9wRCznw8DahDVhi3nZzB6I+38kGFUIMsMPK6iNfp5lr2ue/AFuTxij3QiSv5jZHIKw4GZl3AEmmtkHZvYWQTe8IHM8kxLXV9KxkqZr2cW7uwXVFJDrjjRK+/p24H8Iq7avnXA/i/BAflXSIIIcb4Gk5C2EB/pwwhqqRxVnYGavxObKCEKtZC2CouJCM/tAkoBrzOwnKeX7xNqfgmJp31qTJiks4PtmlqXoWPzkmiqXGS4laZy8/qU0j8ppISUleNsSx22UuL7WnSVx8a9OjcBY4Ewzm1nknpSyHZmRxtVERT4zm1UizEMxzH10lGu9Bzgw1oqQtJakshKwlJemXepnZu8B/yz0sQDfBCaXi5PBBOC7klaIZd1M0qop4TZUlMMl1PTup7zMcN78i0lKHG8GbEiQ4y3l3pK0co2mIQyNmc03s9+keJ0PnCPpAcJCzOXSeAN4GriqTLAphD6PuQRVxLWiW0Fb6OeEzuUZBJH6/hlFLyfFW+x3JKGjdwZBt/zMCtNLciWh+fdYbFZeTnot4GngyJjnWsDvrLzM8NXAZUWdwXm4FOgZm2LjgZFmtqiMe+thBm05tialaSRx4xeVmcDnYg3CaWLq3XTqrCTuRptsaqf+T9q7dFmO/+qXy0riAigoT/6G8PK90szOLfIfCYyhvTVwcVfL4zZKH02nkDSc0Py6wI2M04gY0FaFGksccnEJYczZfMLHi9tjjTzJeDM7odMZVommMDRm9ndC+99xGpPqTUHYAZhrZi8ASBoH7E9oSjcsDdFH4zitgLVZ5kbU3k5sxxYlMwB4OXE8P7oV87XEQM0NUvzrSlPUaByn8cn9VSlLezutr6g44TuAG8xskaTjCOO2Pp+vnLXBazSOUyeq9Hl7PpCsoQwEXi3K5+3E173fEwaidiluaBynDhSWiaiCoZkGbCrp05JWJAxQvT0ZII54L7AfYZhDl+JNJ8epE7ak853BZrZY0gmEQZs9gbFmNkvSmcB0M7sdOFFhAvFi4B2yB7vWHDc0jlMnqjVmzczupGheoJmdltj/CZA2labLcEPjdEtWWmllBg78TF3ymj9/TucTafIpBlm4oXGcOuGGxnGcmlJYJqJVcUPjOPXAwJp4Yass3NA4Tl3wPhrHcepAC9sZNzSOUy+8RuM4Tk0xozBpsiVxQ+M4dcJrNI7j1Bijra11vzrVfFKlpJ9JmhXXxnhC0o5VSne55WK7miigdnGK+zAFIblK0lomjqSroxhcQ9Cd71NVqd6kym5JTWs0cQX+fQnr+C5S0HFescZ59jSzJbXMo4YMAxYSdKZqGacmSOqVoqflFGjhPppa12j6ExbyWQRgZgvM7FVIl2WN7pMknaeOMrArSxoXa0bjCQqLRL+Fks6U9DCwc6m0k8Q3/+8kTZT0gqQ9JY2V9LSkqxPhDlWQdH1K0nnR7buSzk+EGSnporhfSlb3W/F8JgO7ppRnEHAccFKMu7ukjSTdE8/5HkkbZsWJXnsoSAi/kKzdSBqldhndM9JumIJE8LPxPvy+UPOS9BUF0bnHJf1d0nrRvVget9x92kfSQwoSuzcpyL0gaZ6kM1QkvdtMhJHBLolbK+4CNogP7qWS9kz4dZBlTfilycB+lyCbOpggoZpczGdV4Ckz29HM7s9IO8mahJXHTiKsSlZQw9xaQSt7feC8GGYIsL2kA4Cbgf9KpHMwMF4lZHXj+iBnEAzM3rTL8S7FzOYBlwEXWpCknQJcDPwhnvN1wG9zxIFg4HeL530uhB85Qc1yh3gu20laRi0znu8vgJ1iOZM/+PuBnSzI1o4DTkn4JeVxU+9TrM3+HBhuQWJ3OvDDRBqlpHebhlZuOtXU0JjZQsKDdizwFuHHODJ676XSsqxpMqx7EGRdCzK6MxLhlwC3JI7LpZ3kjqhAORN4w8xmmlkbMCvmuz0wyczeik2C6wha4G8BL0jaSdLaBI3rBygtq7tjIp1/EzSO8rAzcH3cv5Z2SdssbjOztrgy/nrRbZ+4PU7QtPoMHWV0dwAmm9k7ZvYJcFPCbyAwIV7TUSx7TZPyuKXu004EA/tAvDZHAkmBvlLSu0tRQhJ3yZJu1kJrce3tmn91iv0lk4BJ8SE9UmHl9nKyrKVkWEuZ/I8L/TKqTPI1KcNaLNHai7BwUCnGEyR15wC3mpnFJloHWd1YC6rG6ypvGslzUeL/OWZ2eZl45bSLLiLI2dwuaRgwOuFXLE+cVk4R9NIPLZF+qXvenmhCErd371W63eu/mWssWdS0RiNpc0nJt+YQgtB7OVnWUiTlVbcCBpcItzxpl+JhYE9J/WJfy6G0S9n+CTgguhVqKKVkdR8GhklaW0HG9qAS+RVL0j5IWKoRwrnfnyNOKSYARyX6RQYUypngkXi+a0rqBXwt4ZeUJz6yTD6l7tNUYFdJm0S/VRRkcluCwuztVm061bpG0we4SFJfQu1gLnCsmb0rqSDLOo9lZVlL8TvgKgV51ycIP4oOLGfaqZjZa5J+AkwkvJHvNLM/R79/SpoNbGFmj0S32ZIKsro9gE+A75nZ1Fizegh4jdB0SZP4vQO4WdL+wPeBE4GxkkYRmp7fyhGn1LncFfuQHgoVLxYC3wDeTIR5RdLZBMP4KkErqCDINxq4SdIrBKPx6RJZpd4nM3srNptvkLRSDPtz4NlSZW4qCr3BLUrTSOI61UFSHzNbGGs0txLWpL21q8tVTO/eq1g9V9j7+OMPOyWJ23/gRnbUiT/LDHf2j7+TKYnbHXEVBKeY0bGz9ingReC2Li5P02Bt2VseJI2Q9IykuZJOLRPuQEmmBhgw6VMQnGUws6b8tNzlGFWZgqCc2tuSViM0vR/udKZVwGs0jlMHqtgZvFR7Ow6VKGhvF3MWcD7wcdVOohO4oXGcOlElQ5OpvS1pW2ADM/tL9UrfObzp5Dh1wfKuR9NP0vTE8RVx/FCBstrb8WvnhTSAaFwSNzSOUw8s94C9BRlfnbK0t1cDtiIMkAX4FHC7pP3MLGnA6oobGsepF9UZSrJUe5swgPIQ4LD2LOw9oF/hWNIk4OSuNDLghsZx6oIBbVVYJsLyaW83HG5onG7JVlttwfTp9XlJDx1ahWEoVVwz2DK0t4vch1Ul007ihsZx6kJzz2XKwg2N49QJNzSO49QcNzSO49QUc+1tx3HqQQtXaNzQOE598M5gx3HqgBsax3FqS/4pCE2JGxrHqQNG9QbsdUdqskyEGlQGV9IBkjpoKlUbJSRvo8BaRYtJSVqY4tZX0vEVprNMHAX53KovHSBpHbWLy+0u6SAFIb6JkoZK+m12Ks2OYW1tmVuzUnVDo2VlcAcDw1l2/YyqE1cdy8MBpIi3xTQavXbXF6jI0CxnnOXhC8AcM9s2itgdDRxvZnuZ2XQzO7EOZWhsWlx7uxY1moaUwZW0C7AfMCbWsjaO+Z6tIFP7A6XIvkrqoSDZ2jeR1tzot46kW2K+0yR1kLotKsPGkv4m6VFJUxSlXyV9WkEqdpqks0pEPxfYOJZ9jAJj4vnOlHRwVpzo1kfSzZLmSLoucQ+2kzQ5lm2Cgrpmcfk7SPRKGkJYye1LMZ/TCUJ3l8XyLa1FSeoj6apY3hmSvhbdU6Vymw2XxK0uDSmDa2YPArcDoyzIxz4fvfqa2Z5m9mtSZF+jcuWfga8CKDQD55nZG8BvCHK02xM0kK7MuDZXAN83s+0Isq+XRvffAL+L6bxeIu6pwPOx7KMIkrxDgG0ItcYxKcahOA7AtoRrvAVBRXNXBa2pi4ADY9nGEq53MR0kes3sCeA0YHzM5wyC3O3hiTwL/AJ4z8y2jmncq2yp3KbB2ixza1aq3lyIUh3bAbsDexFkcE81s6sJUrWnAKsAaxGkZ++IUUvJ4P42pjtDQSuoQJoMbqm0y5GUpx0Yy9sfWJGgAlAIcxpt6egPAAAR4UlEQVRwFWH9j0Kc4cAWicrT6gqLQncgvqV3IWgjFZwL+ka70i7Wdi1B7zuL3YAbLCh0vhFrZdsTjGk5HjGz+bFMTxCu9buExZLujmXrSdCfKmZn2jXHryXUZCphOO2CeAVtrH1pl8qFcN0fSoss6ViCvDIbbrhhhVl3LYU1g1uVmvRLWGPL4BaTlHMtJfv6ELCJpHUI/Ty/jO49gJ2tXXeaWJ60fHoA75rZkBLlqPQpXF6doaRcbuFaC5hlZjtXmNbylLk4TpZUbntmCUncoUOHdq9fbYt/3q5FZ3Ajy+Bmycemyr5aeEJuBS4Anjazt6PXXcAJhXCxvyIVM3sfeFHSQTGsJG0TvR9gWenbPGW/DzhYUs9oAPego3pnXrncZ4B1FDrykbSCpC1TwuWR6C1H8fVak5aRyjXa2toyt2alFn00fYBrJM2OTZ0tgNFm9i5QkKq9jfwyuH1iOqdQRgY3Z9rjgFGxs3fjFP/RhKbNFGBBkd94goRssql1IjA0dmzOBo7LOJ/DgaMlPUlo2hVkMn4AfE/SNIKx60A0bg/Ezt8xBMM3A3gSuJfQn/R6RpxUomzHgcB5sWxPEJp5xZwIfCvej2/GclfCL4E1Y3meBPYys7cIC2nfENOdCtRHgrLOtHIfjUviOt2SoUOHWj1X2Js+fXqnJHH79Vvfvrz/tzPD/WHsWU0pidvoY0ccpykIncFdXYquww2N49SJVm49uFKl49QDM9qWtGVueZA0QtIzCgNHT03xPy4OinxC0v2qw7SbLNzQOE6dqMYUBIXpNpcAXyR8aDk0xZBcHwdFFkZtX1Dtc6kUNzSOUwcKA/aqMNdpB2Cumb0QvxaOo/3rZcgrDKUosCqVj3eqOt5H4zh1IqchydLeHsCyk5TnAx1WR5D0PcJUjhWBz1de2urihsZx6kLuWZNZ2ttpn9k7JGxmlwCXSDqMMJfsyA6x6og3nRynHhhYW/aWg/nABonjgcCrZcKPI0yb6VK8RuN0Sx599NFSc8oalipNMZgGbCrp04TpMocAhyUDSNrUzJ6Lh18GnqOLcUPjOHWgWrO3zWyxpBOACYRZ9mPNbJakM4HpZnY7cIKk4cAnwD/p4mYTuKFxnPpQxdnbZnYncGeR22mJ/UrnoNUcNzSOUxeae9JkFm5oHKdetPAUBDc0jlMnrOvHzXUZbmgcpw6YGW1tS7q6GF2GGxrHqROtPHvbDY3j1Ak3NI7j1Bw3NI7j1JQwO7t5Fx/PInOuk6QlcQGdWZKelPRDScs9R0rSlVkL8aiExnaeuI2KEnrcRe7DFFQ0K0lrmTiSrpaUR1WiVHodtL4zwndKW7xVMWvL3JqVPDWajwpaRJLWBa4nrNR/etlYJTCz7BWaaxA3iaReZra4GmlVgWHAQoKUSS3jOF1MKzedKqqZmNmbBKXAE6Iu0SAFDenH4rYLLH3jTlK6xvPS2ooq1FwuintoXK7wKUnnJcIsTOwfKOnquH+1pAskTSTIioyWNDam+YKkExPxvqGgA/6EpMujdtLRki5MhDlG0gWlwkf3bylIA08mqFEWn88ggkTLSTHu7krRt86KE732kPRgPJcDE+FHKWh6z5B0Rplr+6tYY50qab3o1kGLvEz8dSU9Gve3kWSFskt6XkGvqZS2+XMK2lTE47kKUrlNRZUWvuqWVNwEMrMXYrx1gTeBvaNm8sFE+dpIB43nZDrqhOaypPUJsrGfJwjUbS8pz1T4zWJ+P4rHnwH+k7Bq2ekKwmmfjeeya6zJLSHoMY0D9lPQqQb4FnBVqfAKsrpnxPPeO16HZTCzecBlBP3uIWY2hRR96xxxAPoTZHL3Bc6N12kfYNN4fkOA7STtkXJdVgWmmtk2BGG6Y6J7By3yUhc2voR6S1qdIIc8Hdhd0kbAm2b2YVp6Udv8j7QL5w0HnjSzYl0tJB0rabqWXRiqm2DedFoOCvPzVwAuVlBoXEL4IRdI03hOKhvuRE7N5RS2ByZF8TEkXUdQarwtI95NBRndyF/NbBGwSNKbwHrAF4DtgGmxXCsTfij/knQvsK+kp4EVzGymwkzaDuEJq54lyzi+6PqUYnn1rW+LP9rZiZrHPnF7PB73IRie+4ri/hv4S9x/lGAYobQWeSkeJBjWPYCzgRGEZ6VgDEulNxb4M/C/wFEEjfMOJCVxJXWr17+1uCRuxYZG0n8QjMqbhH6aN4BtCLWcjxNB0zSel0mKnJrLacUo45e8m8X62/8qOi6lQ32Nmf0kJe0rgZ8Cc2j/MaSGjzWsajxZedNInosS/88xs8sz4n5i7b+C5L0qpUVeiimE2sxGBMPxY0L5C0YsNb2ol/6GpM8TDHQpWeBuTSsbmoqaTrEdfRlwcXww1wBei2/SbxLWx8hLZzSXHwb2lNQv9occCkyOfm9I+qzCl7GvVlCeAvcAByp0fCNprVj9x8weJqxudhhwQ0b4h4FhktaOza2DSuRXrI+dR986r6b2BOAoxb4vSQMK5cxJqhZ5Ge4jyAY/F5+Jd4AvEbTFs9K7ktCEurGo1tkkGNbWlrk1K3kMzcqx03EW8HeCUHuhU/FS4EhJUwnNguIaQ0k6oblsZvYa8BNgIkF7+jEz+3P0P5XwBr0XeC1veRKJzyb0Hd0Vy3U3of+jwI3AA2b2z3LhYxlHE5qDfwceK5HlHcBXEx27efSti+OUOpe7CF8JH5I0E7iZfAaqwGhKa5Gn5Tcv7haaZvcD7xauVUZ6txOadqnNpmbAaMvcmpVupb0dfyz7mVlWX0Ety/AXQkfsPV1VhmZE4WvihWZW0nAWha/rg2tmnVo3dPXV17ahQ0dkhps48fqm1N7uNouTS7obmNlVRkZSX0nPEsYVuZGpIgpqi7cQaqlNSaEzuFU/b3ebKQhmtnd2qJrm/y75vho5FWJm5xI/yTcvzW1Isug2NRrH6e60tS3J3PKgbO3tH0qanRj0uVHVT6ZC3NA4Tp2oRtNJ+bS3HweGxkGfN5N/LFbNcEPjOPXALN+WTR7t7YlxJDaEr7kDq3ouy4EbGsepA0ZYMzjrLwdp2tsDyoQ/Gvi/5S95deg2ncGO093JOZepX9Fcrivi1IsCubS3IUz2BYYCe+YuZI1wQ+N0S6Qe9O69al3y+vjj3ONQy5D7q9OCjHE0ubS3FZQqfwbsGefzdSluaBynTtRRe3tb4HJgRJxV3+W4oXGcOhD6ejtvaHJqb48hTOe4Ka4o8A8z26/TmXcCNzSOUxeqN2Avh/b28KpkVEXc0DhOvWjhkcFuaBynTrgkruM4NaeV5zq5oXGcOuDa247j1AWv0TiOU3Pc0DiOU3Na2dBUdVKlXD63KqiB5XPL5HOQpKcVBPqQdENcD+UkSWfGIfEtjIG1ZW9NSrVrNC6fW1uG0bjyuUcDx5vZREmfAnYxsy5fcKlRMIO2JjYkWdRsmQiXz21O+dy0aynpNIJK5mWSxhCUMtZNlHNpLUrS9jHvJ+N1WC1eszGJvL9T+s52X1p5zeCarkfj8rnNJZ9b6lqa2ZmEe3K4mY0C9gOeL8oTSSsC44EfWJDfHQ58RKgNvWdm2xNUSI9RmDTYRLgkbq1x+dzmkc9d3mtZYHOC4OA0ADN7P6azDzA4UbtaI+a9jOKFpGMJtWTi9etWNHONJYuaGhq5fG6zyed29tct0sso4PtmNqFcZEtob/fo0bPb/Wpb2dDUrOkkl89tRvncctcyD3OA9SVtH/NYTVKvmPd3C01NSZtJqs+qVnXCdZ2qy8qx6bMCsJhQnb8g+l0K3CLpIIKUbUXyuZJGEuRzV4rOPweezY5qr0kqyOcKuNM6yue+DDxFaC7kxsxmSyrI4fYAPgG+B7wUg9wIDLGEfG5aeDObKmk0oTn4GkE+N80Q3wHcLGl/4PsE+dyxkkYBbxH6grLilDqXu2If0kOxWbKQoKP9ZiJMuWuZiZn9W9LBwEWSVib0zwwn1P4GAY8pZP4WkKcfrRthNKWkeE66lSRuJcjlc5uaHj16Wj2X8mxrW9KpZmPv3qvahht+NjPcc8896pK43QW5fK7TgHjTqckwl891Go7mNiRZNKWhcZxGo1prBndX3NA4Tp1o5RpNU/bROE7jYVhbW+aWB0kjJD0jaa6kU1P891CYqrNYNZhAuzy4oXGcOlENSdw4fukS4IuE0fKHquMqBf8ARhImNTcE3nRynDpRpT6aHYC5cR4hksYB+wOz2/OxedGvYTqF3NA4Th0ojAzOQZb29gDCANMC8wlz5RoaNzROt6RHjx6stNIqdcnr3//+qAqpVE17O23gYMP3MruhcZw6USXt7fmEOXQFBgKvViPhWuKGxnHqRJX6aKYBm8b1el4hTKo9rBoJ1xL/6uQ49SB00mRvmcnYYuAEwoz3p4EbzWyWwrrM+8HSVQznE1YBuFzSrBqeWS68RuM4dcConiSumd0J3FnkdlpifxqhSdUwuKFxnDrRyiOD3dA4Tp3wuU6O49QYq9ZXp26JGxrHqQMVDNhrStzQOE6daGVD01Cft5UQdEvxGxaXxux2KIjnPVXCvaIxEMVxVEI+t4L0UiWFM/J/Ku5323tSf1pbErehDE0LMojKB1stTxynAajG7O3uSsMZGgXGKEiuzoyr5hfoo3Tp3HmSzohrcMyU9JmUdEdKuk3SHZJelHSCpB9KelzSVElrxXBD4vEMSbdKWjNKsjySSGuQpBlxfztJkyU9KmmCgupkwf1JSQ8RlBHSOBfYXUE69iRJvSVdFc/hcUl7ZcWJbutL+puk5yQtFZFTfhnhgxTkaZ9VlM1VCQnjUsQy9433721JR0T3ayUNL5Ve9N8/kc51hYFnzUYrrxnccIaGoLw4hCA0NxwYU/jxUl46d0GUy/0dcHKJtLci1AZ2AH4FfGhm2xJkTo6IYf4A/DjKzM4ETjezp4EVFQTxIMja3qigQ3QRcKCZbQeMjelCEI070cx2LnOupwJTonTshUSDZGZbEzSTrpFULGxXHId4vQ4GtgYOlrSBKpMR7mVmOxCu7enRrZyEcRoPEO7HlsALQEHneyeCLlep9K4kysRIWgPYhaLBaM2AmdHWtiRza1YasTN4N+CGKEf7hoLo/fbA+5SXzv1T/P8o7TKxxUw0sw+ADyS9R9A8gmBQBscHva+ZFUTRrgFuivs3Al8n1CgOjtvmBON1d6xc9QReS0nnWsJCRXnO/SIAM5sj6SXCIuczMuLdY2bvAUiaDWwE9CW/jHDy2g2K++UkjNOYQpDHfYlg7I+VNAB4x8wWxmvSIT0zmyzpEgWxuv8CbonD7DugZSRxG/EdWZ5mrrFk0YiGppx+Tjnp3EUl3EvFb0sct5WJU2A8cJOkPxGE6Z6TtDUwq7jWIqkvyzd1f3m1g0rJ9eaVEU67didRWsI4jfsINbINgZ8RlD8PJBigrPSuJShtHgIcVSqDpCRur14rdLtfbSsbmkZ8LdxHqP73VJDV3QN4JCNOVYi1gn8W+ikI0r2To9/zhB/iLwhGB+AZYB1JOwNIWkHSllFu5T1Ju8Vwh5fIsliu9r5CWAXJ3w1jHuXilKIzMsJQoYSxmb0M9AM2jau/3U9owhYMTbn0riY02zCzLp8AWCu8j6YBUNBgXgTcSmgqPAncC5xiZq/XsShHEvqFZhD6Ps5M+I0nyMTeCEHilfDWPk/Sk8AThD4GCP0Ol8TO4FIrJ80AFsdO45MIssE9FVQ2xwMjzWxRRpxUzOwtwrqxN8RzmQp06CQvw6XAkZKmEpo5eSSMH6ZdpngKYTW4QtO2ZHpm9gZhJvJVFZSv+1GF2dvdlYaRxJW0DfD72CnptBCSViH0k32u0NeURa9eK9hqq61d24JFPvjgbRYv/qRTkrhBwjd7RcCPPlrokri1QtJxwA2EryROCyFpODAHuCivkemutHLTqWFqNI5TCd2xRrPiisUjFTqyaNGHTVmjacSvTo7ThDR3jSULNzSOUyd8PRrHcWqKLxPhOE4dMK/ROI5Te9zQOI5Tc7zp5DjdjCVLFi949903XqpTdhtVIY0JZtYvR7gFVcir4fBxNI7j1JyGGBnsOE5z44bGcZya44bGcZya44bGcZya44bGcZya44bGcZya44bGcZya44bGcZya44bGcZya8/8BawO7zOHg89UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create example manually (dataset is re-sorted by length)\n",
    "example_id = 8\n",
    "example = test_data.dataset[example_id]\n",
    "X = namedtuple('X', ['story', 'query'])\n",
    "x = X(STORY.process([example.story]), QUERY.process([example.query]))\n",
    "\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    y_hat, attention = model(x)\n",
    "\n",
    "# Plot attention heatmap\n",
    "fig, ax = plt.subplots()\n",
    "cax = ax.matshow(attention[0, :len(example.story)].numpy(), cmap='bone')\n",
    "story = [' '.join(s) for s in example.story]\n",
    "ax.set_title('Q: ' + ' '.join(example.query) + '? A: ' + example.answer[0])\n",
    "ax.set_yticks(torch.arange(len(story)))\n",
    "ax.set_yticklabels(story)\n",
    "ax.xaxis.set_major_locator(ticker.NullLocator())\n",
    "fig.colorbar(cax)\n",
    "display(plt.gcf())\n",
    "clear_output(wait=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
